Function approximation

Not to be confused with Curve fitting.

Several progressively more accurate approximations of the step function.

An asymmetrical Gaussian function fit to a noisy curve using regression.

In general, a function approximation problem asks us to select a function among a well-defined class that closely matches ("approximates") a target function in a task-specific way. The need for function approximations arises in many branches of applied mathematics, and computer science in particular , such as predicting the growth of microbes in microbiology. Function approximations are used where theoretical models are unavailable or hard to compute.

One can distinguish two major classes of function approximation problems:

First, for known target functions approximation theory is the branch of numerical analysis that investigates how certain known functions (for example, special functions) can be approximated by a specific class of functions (for example, polynomials or rational functions) that often have desirable properties (inexpensive computation, continuity, integral and limit values, etc.).

Second, the target function, call it g, may be unknown; instead of an explicit formula, only a set of points of the form (x, g(x)) is provided. Depending on the structure of the domain and codomain of g, several techniques for approximating g may be applicable. For example, if g is an operation on the real numbers, techniques of interpolation, extrapolation, regression analysis, and curve fitting can be used. If the codomain (range or target set) of g is a finite set, one is dealing with a classification problem instead.

To some extent, the different problems (regression, classification, fitness approximation) have received a unified treatment in statistical learning theory, where they are viewed as supervised learning problems.

Polynomial Approximation

Traditional polynomial function approximation schemes like Taylor, Fourier, Pade, Chebyshev, and Remez are deterministic. These schemes generate a polynomial with the same coefficients for a given function and an interval. Trivedi et al. presented a Perceptron (Artificial (Feed-forward) Neural Network (ANN)) based probabilistic polynomial function approximation scheme, where the coefficients of the generated polynomial depend on the Perceptron model hyperparameters such as the learning rate and batch size and randomness of initialized neurons before training.

ANN-based Approximation

Polynomial approximation using ANN.

While Artificial Neural Networks (ANN) are known for their universal function approximation properties, they are often treated as black boxes and used to calculate the output value. Trivedi et al. proposed the use of a basic three-layer Perceptron consisting of an input layer, a hidden layer, and an output layer, with both hidden and output layers having linear activations to generate the coefficients for an approximation polynomial of a given order. In this architecture, the input layer is dynamic, with the input nodes corresponding to the desired polynomial degrees. While having a variable number of hidden layers is possible, they fix it at a single layer with a single node to minimize the computation. They show coefficient calculations for a third-order polynomial ($d=3$); a univariate function $f(x)=y$; and an input $x$, actual output $y$, and predicted output $y_{out}$. The input layer weights are $\{w_{1},w_{2},\ldots ,w_{d}\}=\{w_{1},w_{2},w_{3}\}=\{x,x^{2},x^{3}\}$, and the biases are $\{b_{1},b_{2},b_{3}\}=b_{h}$. Thus, the output of the hidden layer is $y_{h}=w_{1}x+w_{2}x^{2}+w_{3}x^{3}+b_{h}$

The predicted output is calculated by $y_{out}=w_{out}\cdot y_{h}+b_{out}=w_{1}w_{out}x+w_{2}w_{out}x^{2}+w_{3}w_{out}x^{3}+(b_{h}w_{out}+b_{out})$

Where the layer weights $\{w_{1}w_{out},w_{2}w_{out},w_{3}w_{out}\}$ are the coefficients for the approximating polynomial of order three and the constant term is $b_{h}w_{out}+b_{out}$. Since the predicted output ($y_{out}$) is probabilistic, it must be fine-tuned with hyperparameter tuning, as incorrect results lead to erroneous (inefficient) approximations.

See also

• Approximation theory
• Fitness approximation
• Kriging
• Least squares (function approximation)
• Radial basis function network