I am not posting this problem via equations. I am trying hard to find some references related to my problem. Following is a brief description of what I am looking for:
Is there an example of a controlled Markov Chain (CMC) with a finite action space but countably infinite state space where it is possible to find the occupation measure of the various possible actions, by solving finite number of equations. For example if CMC is positive recurrent, I can always say that find the invariant distribution, and to find the occupation measure of an action, just add the probabilities of the states in which that action will be used. But this operation is not possible in finite time, since the state space is infinite.Of course I am not saying that it is always possible to find this, may be it is possible under some stricter conditions or some properties of CMC. Also simple cases like birth death process where action space is null, are excluded.
Any reference will be very useful.