I think (not 100% sure again) that he proves the quotient formula for bounded functions first (and in this special case he probably needs the bounded property) and then extends it in the way above...

I think I meant $I_f = I_g$ at least $X/G$-almost everywhere...

$f(\mathcal{C}g)$, right? The function $\tilde{f}: X/G \mapsto \mathbb{R}, \tilde{f}(xG) := f(x)$ is not well defined. Hence we actually define $I_f(xG) := \int_G f(xg) dg$ which then is well defined. However, this expression needs to be independent of the chosen representative of the $L^1$ class, i.e. given $f=g$ $X$-almost everywhere, is $I_f = I_g$? And I cannot remember why right now but it must be a relationship between the measure on $X$ and the one on $X/G$ that makes this true... that was what I did not understand at the time...

Depending on your teacher, $L^1$ functions might or might not take the value $\pm \infty$ itself. So let's assume that they are allowed to do that. Since they are in $L^1$ we know that the set of $x$ such that $f(x)=\infty$ is a zero set, however, without changing the function on that set you will never achieve that they are bounded. Not 100% sure anymore (It's been a few years :-)) but I think I did not feel comfortable with the relationships of the measures on $X$ and the one on $X/G$: What does the formula say? Well first of all we need to make sense of the expression

Martin L. Puterman, Markov Decision Processes. Unfortunately is not free. In this book, there are proofs for many things like existence of optimal policies, etc. However, he only deals with the value functions $v^\pi$ and $v^*$ and so on. He never actually uses the expression $E[\sum_{k}\gamma^k R_{t+k}|S_t=s,A_t=a] = Q^\pi(s,a)$... That makes me suspicious: Is Q-learning and policy and value iteration all the same? How come there is a whole book on reinforcement learning that strictly proves things but does never mention the $Q$ function? ... weird and confusing...

Did you find an answer to that in the meantime? I am stuck at the exact same point as you are. I am beginning to think that all the Q-function business is actually a 'lie'. First of all the only formal book (by Puterman) abour Reinforcement Learning never mentiones this function... secondly, for many combinations $(s,a)$, $Q^\pi(s,a)$ is actually not senseful: For example, when given a policy $\pi(a|s)=1$ then what is $Q^\pi(s,a')$ supposed to be for $a'\neq a$??? $E[...|S_t=s,A_t=a']$ does not make sense then...

Also, the idea of taking an infinite amount of iid copies of the transition variable is better because it naturally resembles the transitions (the outcome of the $n$-th transition variable is the decision of the finite state automata at the $n$-th point in time) and it also allows us to collect all independence assumptions at once and then define all the state variables at once (in contract to my approach where one needs to assume, define, addume, define, etc). THANKS!!! I understand this much better now :-)

So, essentially the answer is 'yes, assume that $\Delta(s_1,\cdot)$ and $S_1$ are independent'... But I see now that this makes sense: $s_1$ is just any symbol and in fact we want $\Delta(s, \cdot)$ to be independent from $S_1$ (or $S_0$ depending where we start to count) because the 'chance of taking any transition' should not be influenced by the state before... it is always the same independently of what happened beforehand.

For example: they say that $X$ is normally distributed with parameter $\theta = (\mu, \sigma^2)$. They actually mean: We have no clue how $X$ is distributed (and we dont need that knowledge) but we assume that there is a common prob. space on which a RV $X$ and $\Theta$ live, they have a joint density, and $f_{X|\Theta}(x, \theta) = f_{X,\Theta}/f_\Theta = \text{const} \exp(-(x-\mu)^2/2\sigma^2)$.

I was wondering about exactly the same point a while ago (see mathoverflow.net/questions/226438/…) and the answer I came up with is: Yes, you do need to change the old / cook up a new probability space. In that sense Bayesians say that they model $X$ and $\Theta$ but in fact they ASSUME already that there is a joint distribution and that $f_{X|\Theta}=...$ (the function they want).