I am trying to derive the HJB equation in a stochastic setting. Let
me exemplify my problem with the simplest case where there is no control,
just one state variable. Assume a flow payoff is given by
$$
W(X_{t})\equiv E_{t}\left\{ \int_{t}^{\infty}e^{-\rho(s-t)}u(X_{s})ds\right\} 
$$
where $X_{t}$ is given by
$$
dX_{t}=\mu(X_{t},t)dt+\sigma(X_{t},t)dZ_{t}
$$
and $Z_{t}$ is the standard Brownian Motion. For any $dt>0$ we can
write:
$$
W(X_{t})=E_{t}\left\{ \int_{t}^{t+dt}e^{-\rho(s-t)}u(X_{s})ds+e^{-\rho dt}W(X_{t+dt})\right\} 
$$


$$
\left(1-e^{-\rho dt}\right)W(X_{t})=E_{t}\left\{ \int_{t}^{t+dt}e^{-\rho(s-t)}u(X_{s})ds+e^{-\rho dt}\left[W(X_{t+dt})-W(X_{t})\right]\right\} \tag{1}
$$


From Ito calculus we get that (and assuming that $W(\cdot)$ is well behaved):
$$
W(X_{t+dt})-W(X_{t})=\int_{t}^{t+dt}W'(X_{s})dX_{s}+\frac{1}{2}\int_{t}^{t+dt}W''(X_{s})d[X_{s}]=\int_{t}^{t+dt}W'(X_{s})dX_{s}+\frac{1}{2}\int_{t}^{t+dt}\sigma(X_{t},t)W''(X_{s})ds
$$
where the last equality follows from the known properties of the quadratic
variation of the process $X_{t}$. Plugging this back in (1):
$$
\left(1-e^{-\rho dt}\right)W(X_{t})=E_{t}\left\{ \int_{t}^{t+dt}e^{-\rho(s-t)}u(X_{s})ds+e^{-\rho dt}\left[\int_{t}^{t+dt}W'(X_{s})dX_{s}+\frac{1}{2}\int_{t}^{t+dt}\sigma(X_{t},t)W''(X_{s})ds\right]\right\} 
$$


Dividing both sides by $dt$ and taking the limit $dt\rightarrow0$:
$$
\rho W(X_{t})=E_{t}\left\{ u(X_{t})+\lim_{dt\rightarrow0}\frac{\int_{t}^{t+dt}W'(X_{s})dX_{s}}{dt}+\frac{1}{2}\sigma(X_{t},t)W''(X_{t})\right\} 
$$
where I used the fact that when dealing with the Riemann integral:
$\lim_{dt\rightarrow0}\frac{\int_{t}^{t+dt}f(x_{s})ds}{dt}=f(x_{t})$ (from standard
calculus).

As you can see, I am almost there. I just don't know how to deal with
term $\lim_{dt\rightarrow0}\frac{\int_{t}^{t+dt}W'(X_{s})dX_{s}}{dt}$.
For example, assume $\mu(X_{t},t)=0$ and $\sigma(X_{t},t)=1$, so
that $X_{t}$ is simply the standard Brownian Motion $Z_{t}$. In
that case, to get the HJB formula right I would need:
$$
\lim_{dt\rightarrow0}\frac{\int_{t}^{t+dt}W'(Z_{s})dZ_{s}}{dt}=0
$$
But I don't know how to prove that this is true. More generally (for
any $\mu(X_{t},t)$ and $\sigma(X_{t},t)$), I would need to prove:
$$
\lim_{dt\rightarrow0}\frac{\int_{t}^{t+dt}W'(X_{s})dX_{s}}{dt}=\mu(X_{t},t)W'(X_{t})
$$
which I am also not sure how to do. Any ideas?