Value function

The value function of an optimization problem gives the value attained by the objective function at a solution, while only depending on the parameters of the problem. In an economic context, where the objective function usually represents utility, the value function is conceptually equivalent to the indirect utility function.

In a problem of optimal control, the value function is defined as the supremum of the objective function taken over the set of admissible controls. Generally an optimal control problem may be written:

$J(u;\theta )=\int _{t_{0}}^{t_{1}}I(t,x(t),u(t))\,\mathrm {d} t$

where the objective function $J(u;\theta )$ is to be maximized over all admissible controls $u\in U$ for which the corresponding trajectory of $x(t)$, ${\dot {x}}(t)=f(t,x(t),u(t))$. In discrete time, i.e. $t=t_{0},t_{0}+1,\ldots ,t_{1}-1,t_{1}$, the integral would be replaced by a summation in an otherwise identical problem:

$J(u;\theta )=\sum _{t=t_{0}}^{t_{1}}I(t,x(t),u(t))$

The parameters of the problem $\theta =(x_{0},t_{0},x_{1},t_{1})$ are the initial and terminal value of the state variable, $x(t_{0})=x_{0}$ and $x(t_{1})=x_{1}$, as well as the initial time $t_{0}$ and the terminal time $t_{1}$. Then the value function is defined as

${\begin{aligned}V(\theta )&=\sup _{u\in U}J(u;\theta )\\[4pt]&=J(h(x);\theta )\end{aligned}}$

where $h(x)$ is the optimal control or policy function.

By Bellman's principle of optimality, which roughly states that any optimal policy at time $t$, $t_{0}\leq t\leq t_{1}$ taking the current state $x(t)$ as "new" initial condition must be optimal for the remaining problem, gives rise to an important functional recurrence equation, known as the Bellman equation (in discrete time), or Hamilton–Jacobi–Bellman equation (in continuous time). The latter, for the above problem, can be written:

$-{\dot {V}}(x,t)=\max _{u}\left\{\nabla V(x,t)\cdot f(x,u)+I(x,u)\right\}$

where ${\dot {V}}(x,t)$ means the partial derivative of $V$ wrt. the time variable $t$. $a\cdot b$ means the dot product of the vectors a and b and $\nabla V(x,t)$ the gradient of $V$ wrt. the variables $x$. The maximand on the right-hand side is equivalent to the Hamiltonian, with $\nabla V(x,t)$ playing the role of the costate variables.

Although unknown until a solution to the optimization problem is found, the value function itself can be used to find a solution. Benveniste and Scheinkman established sufficient conditions for the differentiability of the value function, which in turn allows the application of the envelope theorem to solve the Bellman equation.