Universal approximation theorem

Universal Approximation Theorem

The universal approximation theorem claims that the standard multilayer feed-forward networks with a single hidden layer that contains finite number of hidden neurons, and with arbitrary activation function2 are universal approximators in C(Rm). Kurt Hornik (1991) showed that it is not the specific choice of the activation function, but rather the multilayer feedforward architecture itself which gives neural networks the potential of being universal approximators. The output units are always assumed to be linear. For notational convenience we shall explicitly formulate our results only for the case where there is only one output unit. (The general case can easily be deduced from the simple case.) The theorem in mathematical terms:

Let φ(·) be a nonconstant, bounded, and monotome-increasing continuous function. Let I_m0 denote the m₀-dimensional unit hypercube [0,1]^m₀. The space of continuous functions on I_m0 is denoted by C(I_m0). Then, given any function f Э C(I_m0) and є > 0, there exist an integer m₁ and sets of real constants α_i, b_i and w_ij, where i = 1, ..., m₁ and j = 1, ..., m₀ such that we may define:

$F(x_{1},...,x_{m_{0}})=$ $\sum _{i=1}^{m_{1}}\mathrm {A} _{i}\varphi (\sum _{j=1}^{m_{0}}w_{i,j}x_{j}+b_{i})$

as an approximate realization of the function f(·); that is,

$|F(x_{1},...,x_{m_{0}})-f(x_{1},...,x_{m_{0}})|<\epsilon$

for all x₁, x₂, ..., x_m0 that lie in the input space.