If I have a process that transitions between states with some set, unknown probability, I can sample to find the transition probability. This probability is a sample average, with a well understood sample distribution.

I can now use this probability to construct my markov chain, but if I simply use the sample average, I have an unbiased estimate of the input, but I don't know how the uncertainty propagates. For instance, if my transition probabilities make it look like this is an absorbing state, but in fact it is only close, I can significantly mis-specify my solution.

I can simulate it, but I think there should be some theoretical work on this, I just don't know where. I am looking for citations, or at least the terms I need to use. I assume some literature exists on this, but I cannot find what or where, because all the terms I search for (sample distribution, etc.) are used differently than I need, referring to the outputs, not the inputs...

  • $\begingroup$ I do not see a clear question here. Could you highlight what you are after precisely? $\endgroup$ – Benoît Kloeckner Oct 25 '13 at 16:39

The problem is that even though you may obtain a unbiased estimate $\hat{S}$ of the stochastic matrix $S$, $\hat{S}^k$ is not an unbiased estimate of $S^k$, the k-steps transition matrix.

To account for the convexity, you need to put a prior on your transition, for instance an independent Dirichlet prior on each column. Observing the Markov chain will give you a Bayesian update to that distribution (which is conveniently a conjugate prior).

You can then sample whole chains, or even marginals within the chain, but every single chain will unfortunately represent a point hypothesis over $S$.

It's important to realize the posterior of the continuation of your chain is in general not Markovian.


Not the answer you're looking for? Browse other questions tagged or ask your own question.