I am trying to derive the HJB equation in a stochastic setting. Let me exemplify my problem with the simplest case where there is no control, just one state variable. Assume the payoff is given by $$ W(X_{t})\equiv E_{t}\left\{ \int_{t}^{\infty}e^{-\rho(s-t)}u(X_{s})ds\right\} $$ where $X_{t}$ is given by $$ dX_{t}=\mu(X_{t},t)dt+\sigma(X_{t},t)dZ_{t} $$ and $Z_{t}$ is the standard Brownian Motion. For any $dt>0$ we can write: $$ W(X_{t})=E_{t}\left\{ \int_{t}^{t+dt}e^{-\rho(s-t)}u(X_{s})ds+e^{-\rho dt}W(X_{t+dt})\right\} $$

$$ \left(1-e^{-\rho dt}\right)W(X_{t})=E_{t}\left\{ \int_{t}^{t+dt}e^{-\rho(s-t)}u(X_{s})ds+e^{-\rho dt}\left[W(X_{t+dt})-W(X_{t})\right]\right\} \tag{1} $$

From Ito calculus we get that (and assuming that $W(\cdot)$ is well behaved): $$ W(X_{t+dt})-W(X_{t})=\int_{t}^{t+dt}W'(X_{s})dX_{s}+\frac{1}{2}\int_{t}^{t+dt}W''(X_{s})d[X_{s}]=\int_{t}^{t+dt}W'(X_{s})dX_{s}+\frac{1}{2}\int_{t}^{t+dt}\sigma(X_{t},t)W''(X_{s})ds $$ where the last equality follows from the known properties of the quadratic variation of the process $X_{t}$. Plugging this back in (1): $$ \left(1-e^{-\rho dt}\right)W(X_{t})=E_{t}\left\{ \int_{t}^{t+dt}e^{-\rho(s-t)}u(X_{s})ds+e^{-\rho dt}\left[\int_{t}^{t+dt}W'(X_{s})dX_{s}+\frac{1}{2}\int_{t}^{t+dt}\sigma(X_{t},t)W''(X_{s})ds\right]\right\} $$

Dividing both sides by $dt$ and taking the limit $dt\rightarrow0$: $$ \rho W(X_{t})=E_{t}\left\{ u(X_{t})+\lim_{dt\rightarrow0}\frac{\int_{t}^{t+dt}W'(X_{s})dX_{s}}{dt}+\frac{1}{2}\sigma(X_{t},t)W''(X_{t})\right\} $$ where I used the fact that when dealing with the Riemann integral: $\lim_{dt\rightarrow0}\frac{\int_{t}^{t+dt}f(x_{s})ds}{dt}=f(x_{t})$ (from standard calculus).

As you can see, I am almost there. I just don't know how to deal with term $\lim_{dt\rightarrow0}\frac{\int_{t}^{t+dt}W'(X_{s})dX_{s}}{dt}$. For example, assume $\mu(X_{t},t)=0$ and $\sigma(X_{t},t)=1$, so that $X_{t}$ is simply the standard Brownian Motion $Z_{t}$. In that case, to get the HJB formula right I would need: $$ \lim_{dt\rightarrow0}\frac{\int_{t}^{t+dt}W'(Z_{s})dZ_{s}}{dt}=0 $$ But I don't know how to prove that this is true. More generally (for any $\mu(X_{t},t)$ and $\sigma(X_{t},t)$), I would need to prove: $$ \lim_{dt\rightarrow0}\frac{\int_{t}^{t+dt}W'(X_{s})dX_{s}}{dt}=\mu(X_{t},t)W'(X_{t}) $$ which I am also not sure how to do. Any ideas?