Mathematics of neural networks in machine learning

A neuron with label $j$ receiving an input $p_{j}(t)$ from predecessor neurons consists of the following components:

• an activation $a_{j}(t)$, the neuron's state, depending on a discrete time parameter,
• an optional threshold $\theta _{j}$, which stays fixed unless changed by learning,
• an activation function $f$ that computes the new activation at a given time $t+1$ from $a_{j}(t)$, $\theta _{j}$ and the net input $p_{j}(t)$ giving rise to the relation

$a_{j}(t+1)=f(a_{j}(t),p_{j}(t),\theta _{j})$,

• and an output function $f_{out}$ computing the output from the activation

$o_{j}(t)=f_{out}(a_{j}(t))$.

Often the output function is simply the identity function.

An input neuron has no predecessor but serves as input interface for the whole network. Similarly an output neuron has no successor and thus serves as output interface of the whole network.

Propagation function

The propagation function computes the input $p_{j}(t)$ to the neuron $j$ from the outputs $o_{i}(t)$and typically has the form

$p_{j}(t)=\sum _{i}o_{i}(t)w_{ij}$.

Bias

A bias term can be added, changing the form to the following:

$p_{j}(t)=\sum _{i}o_{i}(t)w_{ij}+w_{0j}$ , where $w_{0j}$ is a bias.

Neural networks as functions

See also: Graphical models

Neural network models can be viewed as defining a function that takes an input (observation) and produces an output (decision).

$\textstyle f:X\rightarrow Y$ or a distribution over $\textstyle X$ or both $\textstyle X$ and $\textstyle Y$. Sometimes models are intimately associated with a particular learning rule. A common use of the phrase "ANN model" is really the definition of a class of such functions (where members of the class are obtained by varying parameters, connection weights, or specifics of the architecture such as the number of neurons, number of layers or their connectivity).

Mathematically, a neuron's network function $\textstyle f(x)$ is defined as a composition of other functions $\textstyle g_{i}(x)$, that can further be decomposed into other functions. This can be conveniently represented as a network structure, with arrows depicting the dependencies between functions. A widely used type of composition is the nonlinear weighted sum, where $\textstyle f(x)=K\left(\sum _{i}w_{i}g_{i}(x)\right)$, where $\textstyle K$ (commonly referred to as the activation function) is some predefined function, such as the hyperbolic tangent, sigmoid function, softmax function, or rectifier function. The important characteristic of the activation function is that it provides a smooth transition as input values change, i.e. a small change in input produces a small change in output. The following refers to a collection of functions $\textstyle g_{i}$ as a vector $\textstyle g=(g_{1},g_{2},\ldots ,g_{n})$.

ANN dependency graph

This figure depicts such a decomposition of $\textstyle f$, with dependencies between variables indicated by arrows. These can be interpreted in two ways.

The first view is the functional view: the input $\textstyle x$ is transformed into a 3-dimensional vector $\textstyle h$, which is then transformed into a 2-dimensional vector $\textstyle g$, which is finally transformed into $\textstyle f$. This view is most commonly encountered in the context of optimization.

The second view is the probabilistic view: the random variable $\textstyle F=f(G)$ depends upon the random variable $\textstyle G=g(H)$, which depends upon $\textstyle H=h(X)$, which depends upon the random variable $\textstyle X$. This view is most commonly encountered in the context of graphical models.

The two views are largely equivalent. In either case, for this particular architecture, the components of individual layers are independent of each other (e.g., the components of $\textstyle g$ are independent of each other given their input $\textstyle h$). This naturally enables a degree of parallelism in the implementation.

Two separate depictions of the recurrent ANN dependency graph

Networks such as the previous one are commonly called feedforward, because their graph is a directed acyclic graph. Networks with cycles are commonly called recurrent. Such networks are commonly depicted in the manner shown at the top of the figure, where $\textstyle f$ is shown as dependent upon itself. However, an implied temporal dependence is not shown.

Backpropagation

Backpropagation training algorithms fall into three categories:

• steepest descent (with variable learning rate and momentum, resilient backpropagation);
• quasi-Newton (Broyden-Fletcher-Goldfarb-Shanno, one step secant);
• Levenberg-Marquardt and conjugate gradient (Fletcher-Reeves update, Polak-Ribiére update, Powell-Beale restart, scaled conjugate gradient).