Let $X$ be a measurable space, and let $T$ be a measurable transformation $T:X \to X$. Let $\mathcal{P}(X)$ be the space of probability measures on $X$, equipped with the weak* topology. Define the $T$-relative entropy by $h_T :\mathcal{P}(X) \to \mathbb{R}$ by $$ h_T(\nu) = -\int_X\log \frac{dT^{-1}_*\nu}{d\nu}(x)d\nu(x).$$
Is $h_T$ continuous on the set in which it is finite? Is it upper semi-continuous?
Thanks!
Now I'll post some sort of an answer, and not just a comment due to the length.
Your first question is answered above. For the second one, if you're willing to take the minus inside (take the inverse of the Radon-Nykodim derivative), then here's a counter example.
Take $\mathbb{R}/\mathbb{Z}$ and the $\times 2$ map. Obviously, the Haar measure is $T_{2}$-invariant, hence the Radon-Nykodim derivative is $1$ (a.e.) and the relative entropy will be $0$.
Now define the measure $\mu_{2^{n}}$ to be the normalized counting measure over points of the form $p/2^{n}$ in the torus.
Obviously, $\mu_{2^{n}}$ converges (weak-$*$) to the Haar measure, as can be seen by (say) exponential sums calculation.
Now, $T_{2}^{-1}.\mu_{2^{N}}=\mu_{2^{N+1}}$, and $\mu_{2^N} \ll \mu_{2^{N+1}}$, hence the Radon-Nykodim derivative $d \mu_{2^N} / d\mu_{2^{N+1}}$ is basically constant $2$ on the points $p/2^{N}$ appearing in the support of $\mu_{2^{N+1}}$.
Hence $$ H_{T_{2}} (\mu_{2^{N}}) = \int \log{(\frac{d\mu_{2^N}}{d\mu_{2^{N+1}}})} d\mu_{2^N}=\frac{1}{2^{N+1}}\log(2)\cdot 2^{N}$$.
The usage of the inverse Radon-Nykodim derivative is due to the fact that $\mu_{2^N} \ll \mu_{2^{N+1}}$ but not the other way around. Because your formula for relative entropy only makes sense when the induced measure is absolutely continuous wrt the original measure, one should be careful of the formula (notice that the inverse Radon-Nykodim derivative $(d\mu / d\nu)^{-1}$ is equal (a.e.) to $d\nu / d\mu$ if both measures are absolutely continuous with respect to one another).
Also in ergodic theory, one will be interested in invariant measures, for which the Radon-Nykodim derivative will be $1$, therefore it's a bit hard to construct proper examples. I suspect that one can "smooth" this example to a proper counter-example (say by taking suitable bump functions over the points of the form $p/2^{N}$ and glue the bumps together so that this measure will be absolutely continuous wrt the Lebesgue measure, I suspect such a measure will satisfy your conditions).
Notice that even regular entropy function (over invariant measures) is not necessarily upper (nor lower) semi-continuous. It turns out this is true for say expansive systems and some other related systems, see for example in Walters' book.
