The Hamilton–Jacobi–Bellman (HJB) equation is a partial differential equation which is central to optimal control theory. The solution of the HJB equation is the value function which gives the minimum cost for a given dynamical system with an associated cost function.
Contents
- Optimal control problems
- The partial differential equation
- Deriving the equation
- Solving the equation
- Extension to stochastic problems
- Application to LQG Control
- References
When solved locally, the HJB is a necessary condition, but when solved over the whole of state space, the HJB equation is a necessary and sufficient condition for an optimum. The solution is open loop, but it also permits the solution of the closed loop problem. The HJB method can be generalized to stochastic systems as well.
Classical variational problems, for example the brachistochrone problem, can be solved using this method.
The equation is a result of the theory of dynamic programming which was pioneered in the 1950s by Richard Bellman and coworkers. The corresponding discrete-time equation is usually referred to as the Bellman equation. In continuous time, the result can be seen as an extension of earlier work in classical physics on the Hamilton–Jacobi equation by William Rowan Hamilton and Carl Gustav Jacob Jacobi.
Optimal control problems
Consider the following problem in deterministic optimal control over the time period
where C[ ] is the scalar cost rate function and D[ ] is a function that gives the economic value or utility at the final state, x(t) is the system state vector, x(0) is assumed given, and u(t) for 0 ≤ t ≤ T is the control vector that we are trying to find.
The system must also be subject to
where F[ ] gives the vector determining physical evolution of the state vector over time.
The partial differential equation
For this simple system, the Hamilton–Jacobi–Bellman partial differential equation is
subject to the terminal condition
where the
The unknown scalar
Deriving the equation
Intuitively HJB can be derived as follows. If
Note that the Taylor expansion of the first term is
where
Solving the equation
The HJB equation is usually solved backwards in time, starting from
When solved over the whole of state space, the HJB equation is a necessary and sufficient condition for an optimum. If we can solve for
In general case, the HJB equation does not have a classical (smooth) solution. Several notions of generalized solutions have been developed to cover such situations, including viscosity solution (Pierre-Louis Lions and Michael Crandall), minimax solution (Andrei Izmailovich Subbotin), and others.
Extension to stochastic problems
The idea of solving a control problem by applying Bellman's principle of optimality and then working out backwards in time an optimizing strategy can be generalized to stochastic control problems. Consider similar as above
now with
where
Note that the randomness has disappeared. In this case a solution
Application to LQG Control
As an example, we can look at a system with linear stochastic dynamics and quadratic cost. If the system dynamics is given by
and the cost accumulates at rate
Assuming a quadratic form for the value function, we obtain the usual Riccati equation for the Hessian of the value function as is usual for Linear-quadratic-Gaussian control.