MathOverflow is a question and answer site for professional mathematicians. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

If I have a process that transitions between states with some set, unknown probability, I can sample to find the transition probability. This probability is a sample average, with a well understood sample distribution.

I can now use this probability to construct my markov chain, but if I simply use the sample average, I have an unbiased estimate of the input, but I don't know how the uncertainty propagates. For instance, if my transition probabilities make it look like this is an absorbing state, but in fact it is only close, I can significantly mis-specify my solution.

I can simulate it, but I think there should be some theoretical work on this, I just don't know where. I am looking for citations, or at least the terms I need to use. I assume some literature exists on this, but I cannot find what or where, because all the terms I search for (sample distribution, etc.) are used differently than I need, referring to the outputs, not the inputs...

share|cite|improve this question

closed as unclear what you're asking by Benoît Kloeckner, David White, j.c., Andrey Rekalo, Chris Godsil Oct 25 '13 at 18:04

Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question.If this question can be reworded to fit the rules in the help center, please edit the question.

I do not see a clear question here. Could you highlight what you are after precisely? – Benoît Kloeckner Oct 25 '13 at 16:39

The problem is that even though you may obtain a unbiased estimate $\hat{S}$ of the stochastic matrix $S$, $\hat{S}^k$ is not an unbiased estimate of $S^k$, the k-steps transition matrix.

To account for the convexity, you need to put a prior on your transition, for instance an independent Dirichlet prior on each column. Observing the Markov chain will give you a Bayesian update to that distribution (which is conveniently a conjugate prior).

You can then sample whole chains, or even marginals within the chain, but every single chain will unfortunately represent a point hypothesis over $S$.

It's important to realize the posterior of the continuation of your chain is in general not Markovian.

share|cite|improve this answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.