Consider two Markov processes $p$ and $q$. The conditional relative entropy between them is \begin{align} D(p\parallel q)& =\sum_a p(a)\sum_b p(b\mid a)\log\frac{p(b\mid a)}{q(b\mid a)}\\ & =\sum_{ab}p(ab)\log\frac{p(b\mid a)}{q(b\mid a)} \end{align} where $p(a)$ and $p(ab)$ are the first and second-order stationary distributions, respectively. I want to know: is $D$ convex in $p(ab)$?
As usual, $p(a)$ is the eigenvector of the Markov transition matrix $p$ corresponding to eigenvalue $1$. The distributions are related by $$ p(b\mid a)=\frac{p(ab)}{\sum\limits_{b'}p(ab')}=\frac{p(ab)}{p(a)} $$
Note that $D$ can also be written $$ D(p\parallel q)=D(p(ab)\parallel q(ab))-D(p(a)\parallel q(a)) $$ i.e. the divergence between the second-order distributions minus the divergence between the first-order distributions. Sorry for the not great notation.
