1
$\begingroup$

Consider $X_n\in \mathcal{X}$ a controlled Markov chain taking value in a compact set $\mathcal{X}$ with action $a\in \mathcal{A}$, where the action set $|\mathcal{A}|$ is finite.

(In particular, we consider $\mathcal{X}=\left\{\mathbf{x}\in \mathbb{R}^n:\sum_{i=1}^nx_i=1,\ x_i \geq 0, \ \forall i\right\}$)

Specifically, define a class of transition kernel $\mathcal{T_a}:C(\mathcal{X}) \to C(\mathcal{X})$ as

\begin{equation} (\mathcal{T_a} \circ h)(\mathbf{x})=E[h(X_n)|X_{n-1}=\mathbf{x}\ ;a] \ \ \ \forall a\in \mathcal{A}. \end{equation}

Now, given a concave function $\delta: \mathcal{X}\to \mathbb{R}$. Suppose that there exists a bounded solution $(\lambda,h)=(\lambda_{\delta},h_{\delta})\in \mathbb{R}\times C(\mathcal{X})$ to the functional equation

$$ \lambda+h(\mathbf{x})=\delta(\mathbf{x}) + \min_{a\in \mathcal{A}} E[h(X_n)|X_{n-1}=\mathbf{x}\ ;a] $$ and let $$ \gamma_{\delta}(\mathbf{x}):= \underset{a\in \mathcal{A}}{\operatorname{argmin}} E[h_{\delta}(X_n)|X_{n-1}=\mathbf{x}\ ;a]. $$ Clearly

$$(\mathcal{T_{\gamma_\delta}} \circ h)(\mathbf{x}) = E[h(X_n)|X_{n-1}=\mathbf{x}\ ;\ \gamma_\delta(\mathbf{x})]$$ defines a Markov transition kernel, and $(\lambda,h) =(\lambda_{\delta},h_{\delta})$ is a solution to its poisson equation $$ \lambda+h(\mathbf{x})=\delta(\mathbf{x}) + (\mathcal{T_{\gamma_\delta}} \circ h)(\mathbf{x}). $$

Question: Now given another concave function $\delta': \mathcal{X}\to \mathbb{R}$, can we guarantee that the poisson equation $$ \lambda+h(\mathbf{x})=\delta'(\mathbf{x}) + (\mathcal{T_{\gamma_\delta}} \circ h)(\mathbf{x}) $$ has a bounded solution $(\lambda,h)=(\lambda_{\delta'},h_{\delta'})$?

some thoughts: by contraction mapping this equation $$ h(\mathbf{x})=\delta'(\mathbf{x}) + \beta (\mathcal{T_{\gamma_\delta}} \circ h)(\mathbf{x}) $$ has a unique solution $h_{\beta}$ for each $\beta<1$. So if the set of functions $\{h_{\beta}\}_{0<\beta<1}$ is sequentially precompact and $ (1-\beta) h_{\beta}(\mathbf{\bar{x}})$ is bounded for some $\bar{x}$, we can then establish a solution by taking a subsequence of $\beta$ goes to 1 as \begin{equation} \begin{aligned} \lambda &= \lim_{\beta \rightarrow 1} (1-\beta) h_{\beta}(\mathbf{\bar{x}})\\ h &= \lim_{\beta \rightarrow 1} [h_{\beta} - h_{\beta}(\mathbf{\bar{x}})]. \end{aligned} \end{equation} But I don't know how to show the sequential pre-compactness either.

$\endgroup$
4
  • $\begingroup$ What does the notation $E[h(X_n)|X_{n-1}=\mathbf{x}\ ;a]$ mean? If you are referring to a Poisson problem, then the actions are on the $\mathcal{X}$ too, I guess $C(\mathcal{X})$ is compactly supported measurable functions? $\endgroup$
    – Henry.L
    Apr 26, 2017 at 3:33
  • $\begingroup$ The notation means that the transition probability measure from n-1 to n depend on a. So we have a class of transition kernel indexed by a. C(X) is the cts functions. Since the space is compact, they are indeed compactly supported. $\endgroup$ Apr 26, 2017 at 3:39
  • $\begingroup$ So on which object $\mathcal{A}$ actually acts on....or in this setting it is just an index set?I was thinking that $a$ is pullbacks of translations. $\endgroup$
    – Henry.L
    Apr 26, 2017 at 3:41
  • $\begingroup$ There might be some terminology issue. A is the index set. When I say action I meant that one can choose "a" to "control" the Markov chain for e.g. Minimizing some cost. I am not sure what is pullback translation. In short, A is simply the index set of the index for the class of the Markov chain transition kernel. Btw, this is the poissob equation that I was referring to ams.sunysb.edu/~feinberg/MDPHandBook/9.pdf $\endgroup$ Apr 26, 2017 at 3:44

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.