3
$\begingroup$

If I have a process that transitions between states with some set, unknown probability, I can sample to find the transition probability. This probability is a sample average, with a well understood sample distribution.

I can now use this probability to construct my markov chain, but if I simply use the sample average, I have an unbiased estimate of the input, but I don't know how the uncertainty propagates. For instance, if my transition probabilities make it look like this is an absorbing state, but in fact it is only close, I can significantly mis-specify my solution.

I can simulate it, but I think there should be some theoretical work on this, I just don't know where. I am looking for citations, or at least the terms I need to use. I assume some literature exists on this, but I cannot find what or where, because all the terms I search for (sample distribution, etc.) are used differently than I need, referring to the outputs, not the inputs...

Improve this question
$\endgroup$
1
  • $\begingroup$ I do not see a clear question here. Could you highlight what you are after precisely? $\endgroup$
    – Benoît Kloeckner
    Commented Oct 25, 2013 at 16:39

1 Answer 1

Reset to default
1
$\begingroup$

The problem is that even though you may obtain a unbiased estimate $\hat{S}$ of the stochastic matrix $S$, $\hat{S}^k$ is not an unbiased estimate of $S^k$, the k-steps transition matrix.

To account for the convexity, you need to put a prior on your transition, for instance an independent Dirichlet prior on each column. Observing the Markov chain will give you a Bayesian update to that distribution (which is conveniently a conjugate prior).

You can then sample whole chains, or even marginals within the chain, but every single chain will unfortunately represent a point hypothesis over $S$.

It's important to realize the posterior of the continuation of your chain is in general not Markovian.

Improve this answer
$\endgroup$

Not the answer you're looking for? Browse other questions tagged .