Timeline for Proof of Bellman optimality equation for finite Markov Decision Processes
Current License: CC BY-SA 4.0
12 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Mar 9, 2021 at 18:53 | answer | added | user240643 | timeline score: 0 | |
| Jul 6, 2019 at 15:52 | answer | added | 王子銘 | timeline score: 2 | |
| Apr 9, 2019 at 15:02 | comment | added | Fabian Werner | Martin L. Puterman, Markov Decision Processes. Unfortunately is not free. In this book, there are proofs for many things like existence of optimal policies, etc. However, he only deals with the value functions $v^\pi$ and $v^*$ and so on. He never actually uses the expression $E[\sum_{k}\gamma^k R_{t+k}|S_t=s,A_t=a] = Q^\pi(s,a)$... That makes me suspicious: Is Q-learning and policy and value iteration all the same? How come there is a whole book on reinforcement learning that strictly proves things but does never mention the $Q$ function? ... weird and confusing... | |
| Apr 9, 2019 at 11:24 | comment | added | hardhu | No, I gave up and moved forward reading Barto and Sutton book where I found some other obscure parts so I didn't try anymore to be very formal in approaching this subject...by the way, which Puterman's book are you exactly referring to? | |
| Apr 8, 2019 at 10:04 | comment | added | Fabian Werner | Did you find an answer to that in the meantime? I am stuck at the exact same point as you are. I am beginning to think that all the Q-function business is actually a 'lie'. First of all the only formal book (by Puterman) abour Reinforcement Learning never mentiones this function... secondly, for many combinations $(s,a)$, $Q^\pi(s,a)$ is actually not senseful: For example, when given a policy $\pi(a|s)=1$ then what is $Q^\pi(s,a')$ supposed to be for $a'\neq a$??? $E[...|S_t=s,A_t=a']$ does not make sense then... | |
| Feb 1, 2019 at 22:51 | answer | added | waynemystir | timeline score: 2 | |
| Jan 27, 2019 at 11:46 | answer | added | hardhu | timeline score: 3 | |
| Jan 26, 2019 at 20:28 | history | edited | YCor |
edited tags
|
|
| Jan 26, 2019 at 20:17 | answer | added | Sotiris | timeline score: 3 | |
| Jan 26, 2019 at 17:17 | history | edited | hardhu | CC BY-SA 4.0 |
added 23 characters in body
|
| Jan 25, 2019 at 16:10 | review | First posts | |||
| Jan 25, 2019 at 16:11 | |||||
| Jan 25, 2019 at 16:06 | history | asked | hardhu | CC BY-SA 4.0 |