1
$\begingroup$

Let $x = (x_1,x_2) \in \mathbb{N}^2$ be the state, $u$ be the control, and the dynamics be given by $x^{(k+1)} = f(x^{(k)}, u^{(k)}, w^{(k)})$ where $w^{(k)}$ is an IID noise source. For some stage cost $g(x,u)$, the optimal cost-to-go is defined by

$J(x) = \min_u E[g(x,u) + J(f(x,u,w))]$

Suppose I can show that there is a monotone optimal policy. Now for a fixed parameter $\alpha \in [0,1]$ consider the modification

$J(x_1, x_2) = \min_u \{\alpha E[g(x_1, x_2,u) + J(f(x_1, x_2,u,w))] + (1 - \alpha) E[g(x_1 + 1, x_2,u) + J(f(x_1 + 1, x_2,u,w))]\}$

Numerical computation shows me that for my specific problem, that there is again a monotone optimal policy. Is this necessarily the case? Could one prove this in general?

The intuition is that we flip a coin (independently of everything else in he model) and with probability $\alpha$, the state evolves as it did in the original problem. With probability $1 - \alpha$, $x_1$ is incremented. It seems reasonable to me that one could prove that the modified system would have the same structural properties as the original one, but I'm having trouble showing this.

$\endgroup$

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.