Skip to main content
2 of 2
added 17 characters in body
hardhu
  • 171
  • 1
  • 6

Since the characters allowed for comments are not enough to express my doubts about your answer, I write them here. I agree with you that one possible way to answer to my question by contradiction is the one you have provided, and intuitively this idea works, but I still believe that it should be formalized correctly. Let me be more precise: what you are writing is that if there exists $\bar{s} \in S$ such that $v_*(\bar{s}) < \max_{a \in A} q_*(\bar{s},a)$, then taking an optimal policy $\pi_*$ and $a_{\bar{s}} \in \text{arg}\,\max\limits_{a \in A}q_*(\bar{s},a)$ we can define a policy $\pi_{\bar{s}}$ such that $$\pi_{\bar{s}}(a|s) = \begin{cases} \pi_*(a|s) \qquad \text{if} \qquad s\not= \bar{s}\\ 1 \qquad \text{if} \qquad s= \bar{s}, a=a_{\bar{s}}\\ 0 \qquad \text{otherwise} \end{cases} $$

Then it is clear that for that policy $v_{\pi_{\bar{s}}}(\bar{s})=q_{\pi_{\bar{s}}}(\bar{s},a_{\bar{s}})$, so that if we prove that $q_{\pi_{\bar{s}}}(\bar{s},a_{\bar{s}})=q_*(\bar{s},a_{\bar{s}})$ we are done since we get the contradiction $v_{\pi_{\bar{s}}}(\bar{s}) > v_*(\bar{s})$.

But my doubt is then exactly this: why is it true that $q_{\pi_{\bar{s}}}(\bar{s},a_{\bar{s}})=q_*(\bar{s},a_{\bar{s}})$?

hardhu
  • 171
  • 1
  • 6