Timeline for Proof of Bellman optimality equation for finite Markov Decision Processes
Current License: CC BY-SA 4.0
14 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Sep 23, 2023 at 19:28 | comment | added | piero | @waynemystir The way you defined $P_{\phi_{t}}$ does not define a probability distribution, so $q_{\phi_{t}}$ does not exist. | |
| Feb 5, 2019 at 23:29 | comment | added | waynemystir | I don't see that he makes any such restriction to stationary policies after chapter 2. And the similar equation to which your question refers is in chapter 3. | |
| Feb 5, 2019 at 15:56 | comment | added | hardhu | Sorry, I wasn't very clear in my comment: you are right, from a general point of view there is no reason not to consider time-dependent policies, and this instead could be an advance in tackling non-stationary problems. What I meant is that in the description of Markov decision process in Sutton and Barto book which I mentioned, policies were introduced as dependent only on states, since the aim there is to find a rule to choose the best action in a state regardless of the time step in which the state is visited. | |
| Feb 4, 2019 at 17:07 | history | edited | waynemystir | CC BY-SA 4.0 |
correct MDP.5 from a_0 to a
|
| Feb 3, 2019 at 22:04 | history | edited | waynemystir | CC BY-SA 4.0 |
time -> time step in a few places
|
| Feb 3, 2019 at 21:47 | history | edited | waynemystir | CC BY-SA 4.0 |
add definition of policy as set of time-dependent PMFs
|
| Feb 3, 2019 at 21:42 | comment | added | waynemystir | This looks like a great book, thanks. In regards to your doubt, can you think of a reason why the policy should not be dependent on the time step? There's no mathematical reason to preclude the definition of a policy as time-dependent. I will add a quick blurb about this to the beginning of my answer. I would really appreciate your further thoughts on this. | |
| Feb 3, 2019 at 11:03 | comment | added | hardhu | Only doubt I have with regard to this proof is that usually policies are not defined with respect to a particular time step, but only w.r.t. the states (that is $\pi(a|s)$, not $\pi(a|s,t)$). It reminds me the proof given by S. Ross at page 31 of his book "Introduction to stochastic dynamic programming" which I unsuccessfully tried to modify in order to apply to this case. Thanks for your contribution. | |
| Feb 2, 2019 at 4:34 | history | edited | waynemystir | CC BY-SA 4.0 |
fix time-dependent policy definitions
|
| Feb 2, 2019 at 2:21 | history | edited | waynemystir | CC BY-SA 4.0 |
fix typo
|
| Feb 2, 2019 at 2:07 | history | edited | waynemystir | CC BY-SA 4.0 |
remove unnecessary items
|
| Feb 1, 2019 at 23:01 | history | edited | waynemystir | CC BY-SA 4.0 |
added 52 characters in body
|
| Feb 1, 2019 at 22:55 | review | First posts | |||
| Feb 2, 2019 at 0:04 | |||||
| Feb 1, 2019 at 22:51 | history | answered | waynemystir | CC BY-SA 4.0 |