1
$\begingroup$

I am currently working on the optimal control of certain classes of stochastic processes and I have difficulties understanding the roles of verification theorems.

My problem is the following: I am not sure to understand whether this is purely a problem that arises from the use of partial differential equations for which we may need to consider viscosity solutions or whether this is something related to the connection between the optimal control problem and its solution expressed in terms of the value function which is a solution to the Hamilton-Jacobi-Bellman (HJB) equation, which is a PDE in many instances of those problems. Would not be the Bellman optimality principle ensures that the optimal control can be computed from the value function itself a solution of the HJB equation?

I am also wondering whether this a specificity of stochastic optimal control problems because I am not sure to have seen verification theorems in the deterministic setting.

Thanks and feel free to comment to ask for more details.

$\endgroup$

1 Answer 1

2
+100
$\begingroup$

On the role of verification theorem: it is an issue related to the existence-uniqueness of solutions in the classical sense for the HJB PDE. In applying the verification theorem, we ignore such issues, guess the structure of a smooth value function, formally verify (by substitution) that the guessed structural form satisfies the HJB PDE under consideration, and then use the Bellman's principle of optimality to compute the optimal control. Whether such verification is valid remains contingent on the existence-uniqueness of smooth enough classical solution (at least $C^1$ in the deterministic case and $C^2$ in the stochastic case) for the HJB PDE.

On deterministic versus stochastic: The above verification/HJB classical solution issue is for both the deterministic and the stochastic case. For example, see Ch. 4, Sec. 2 of [1], which specifically talks about verification theorems for the first order HJB PDEs in deterministic optimal control. Example 2.3 there is about an 1D deterministic optimal control problem whose HJB PDE does not admit any $C^{1}([0,T],\mathbb{R})$ solution. Ch. 4 and Ch. 5 of that book discusses details on the verification theorems for both the deterministic and stochastic optimal control problems, and also the viscosity solutions.

[1] J. Yong and X.Y. Zhou, Stochastic Controls: Hamiltonian Systems and HJB Equations, vol 43. Springer, New York, 1999.

$\endgroup$
5
  • 1
    $\begingroup$ Thanks for the taking the time to write down this answer. Let me check the book you are referring to and I will get back to you as soon as possible. $\endgroup$
    – KBS
    Commented Dec 27, 2022 at 17:11
  • $\begingroup$ Alright, so I have checked the resources you mentioned and this is now much clearer. Just one last question. Assuming, for instance, that the HBJ is not a PDE but a differential equation in time and the space variable is discrete. Would a verification theorem be needed here? What if we can prove existence, uniqueness and, hopefully, differentiability of the solutions? $\endgroup$
    – KBS
    Commented Jan 1, 2023 at 18:09
  • $\begingroup$ The verification theorem would still be needed in general. If we can prove existence, uniqueness and required smoothness separately, then those + verification by substitution, together will allow us to completely determine the value function, and thereby the optimal control. Hope this helps. $\endgroup$ Commented Jan 1, 2023 at 19:08
  • 1
    $\begingroup$ Yes, that helps. I have validated your answer and gave you the bounty. $\endgroup$
    – KBS
    Commented Jan 1, 2023 at 19:43
  • $\begingroup$ If you want, I have the same question open there math.stackexchange.com/questions/4604971/… and I have no valid answer. If you feel like it, you can copy/paste your answer and can award you the bounty there as well. If your are not interested, I can answer myself with a link to your answer. $\endgroup$
    – KBS
    Commented Jan 2, 2023 at 17:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.