Deriving the HJB equation for exponential utility

Question

I would like to derive the HJB equation for the following stochastic optimal control problem:

$ \Phi(t,x)=\sup_{h} E \left[\exp \left\{\gamma \int_t^T g(X_s,h(s);\gamma)\ ds \right\} \right]$

where the state variable $X(t)$ is given

$dX(t)=\mu(X(t),h(t),t))\ dt + \sigma(X(t),h(t),t))\ d W(t) $

$g$ is a known function and $\gamma$ is just a constant different from $0$.

Honestly I never studied this topic formally and doing some research on the internet I read that the objective functional is usually written as an expectation of an integral over time. Splitting the integral in two, the Bellman principle can be readily applied and this yields the HJB equation.

In this case, however, we have the expectation of an exponential of an integral and quite frankly I am really not sure how to proceed.

Could you please help me? Thanks a lot :)

P.S. I will try to be more concrete: Let us consider the following problem

$ J(t_0,x_0)=\max_{u} E \left[ \int_{t_0}^T f(t,x,u)\ dt \right]$

subject to

$d x(t)=g(t,x(t),u(t))\ dt + \sigma (t,x(t),u(t))\ d B_t$.

Exploiting the linearity of the integral, we can write

$ J(t_0,x_0)=\max_{u} E \left( \int_{t_0}^{t_0+\Delta t} f(t,x,u)\ dt + \int_{t_0+\Delta t}^T f(t,x,u)\ dt \right)$.

Using the Bellman's principle and the law of iterated expectations, we can write

$J(t_0,x_0)=\max_{u, t_0 \leq t_0+\Delta} E \left[f(t,x,u) \Delta + J(t_0+\Delta,x_0+\Delta x) \right]$.

Applying Ito's lemma and taking the limit $\Delta t \to 0$, we derive the following HJB equation:

$-\frac{J (t,x)}{\partial t} = \max_{u} \left(f(t,x,u)+g(t,x,u) \frac{\partial J(t,x)}{\partial x} + \frac{1}{2} \sigma(t,x,u)^2 \frac{\partial^2 J}{\partial x^2}\right)$.

My question is the following: In order to derive the HJB equation for the exponential case, could we apply a similar reasoning? Since the exponential is not additive, it is not clear to me how to do it.

I found a paper where the author solves an exponential control problem claiming (but not proving) that the HJB equation is (up to a minus or plus sign)

$ \frac{\partial J}{\partial t} + sup (\frac{1}{2} \sigma^2 \nabla^2 J + \mu \nabla J -\frac{\gamma}{2} \sigma^2 (\nabla v)^2 - g)=0$.

Can someone explain to me where this equation comes from? As you can see, it has some terms in common with the "standard" one, but there is also some "new" term.

Answer 1

This class of problems is called Risk Sensitive Optimal Control. The case with $\gamma > 0$ is called the risk averse case, and the case with $\gamma < 0$ is called risk seeking case, for intuitive reasons. The earliest instance of this problem was solved by Jacobson in 1973 where the integrand was a quadratic function of state $X(t)$ and control $h(t)$, and the dynamics was a linear stochastic differential equation, i.e., $\mu(X(t),h(t),t) = A X(t) + B h(t)$, $\sigma$ constant, and $W(t)$ as Gaussian white noise. The resulting formulation was called Linear Quadratic Exponential Gaussian (LEQG) optimal control problem, as it was a generalization of well known LQG problem (in the limit $\gamma \rightarrow 0$).

An intuitive way to think of the cost functional is to mentally expand the exponential in Taylor series and think how the sign of $\gamma$ affects the cost-to-go. You can find lot of control-theoretic literature online on this well studied problem, mostly for cases when something is known about the structure of $g$, $\mu$, and $\sigma$.

Answer 2

The formal derivation given below is adapted from the one appearing in Section III.7 of:

• Fleming, W. H., and H. M. Soner. Controlled Markov processes and viscosity solutions. Vol. 25. Springer Science & Business Media, 2006.

Let $\mathcal{C}$ denote the set of admissible controls. Given $h \in \mathcal{C}$, a $C^2$ test function $f(x)$, and $t>0$, define the differential operator $L_t^h$ by: $$ L_t^{h} f(x) = \gamma g(x,h_t) f(x)+ \mu(x,h_t,t) \partial_x f(x) + \frac{1}{2} \sigma(x,h_t,t)^2 \partial_{xx} f(x) $$ Note that the first term in $L_t^h f(x)$ accounts for the exponent appearing in the value function.

By definition of the value function, for any $\Delta>0$ we have that: \begin{align*} \Phi(t,x) &\ge \mathbb{E} \left\{ \exp\left( \gamma \int_t^{t+\Delta} g(X_s, h_s) ds \right) \Phi(t+\Delta, X_{t+\Delta}) \right\} \\ &\ge \mathbb{E} \left\{ \left( \exp\left( \gamma \int_t^{t+\Delta} g(X_s, h_s) ds \right) - 1 \right) \Phi(t+\Delta, X_{t+\Delta}) \right\} + \mathbb{E} \left\{ \Phi(t+\Delta, X_{t+\Delta}) \right\} \end{align*} This inequality can be rewritten as: \begin{align*} 0 &\ge \frac{1}{\Delta} \mathbb{E} \left\{ \Phi(t+\Delta, X_{t+\Delta}) - \Phi(t,x) \right\}\\ & + \frac{1}{\Delta} \mathbb{E} \left\{ \left( \exp\left( \gamma \int_t^{t+\Delta} g(X_s, h_s) ds \right) - 1 \right) \Phi(t+\Delta, X_{t+\Delta}) \right\} \end{align*} Note that the second term appearing in this inequality will contribute a term to ($\dagger$) given below. Pass to the limit as $\Delta \to 0^+$ to obtain $$ \partial_t \Phi(t,x) + L_t^h \Phi(t,x) \le 0 \tag{$\dagger$} $$ which holds for all $h \in \mathcal{C}$.

Now let $h^{\star} \in \mathcal{C}$ be the optimal control, and $X^{\star}$ be the Markov process generated by $L_t^{h^{\star}}$. Then one can write the value function as: $$ \Phi(t,x) = \mathbb{E} \exp\left( \gamma \int_t^T g(X^{\star}_s, h^{\star}_s) ds \right) $$ In this case, a Feynman-Kac formula implies that $\Phi(t,x)$ satisfies: $$ \partial_t \Phi(t,x) + L_t^{h^{\star}} \Phi(t,x) = 0 \tag{$\ddagger$} $$ with the terminal condition $\Phi(T,x)=1$.

Combine ($\dagger$) and ($\ddagger$) to get the requested HJB equation for $\Phi(t,x)$.