0
votes
0answers
17 views

Zeros of non-lipschitz functions (when noisy estimates are available only)

Given noisy (martingale difference) of a Lipschitz continuous function $f$ it is known how to compute zeros of it. It is the stochastic approximation approach (by Borkar, Kushner and Yin etc.). Is ...
0
votes
0answers
19 views

Examples of POMDPs where the actions impact the transitions of the underlying markov Chain

I am not sure if the following is a legitimate question for this board. I am looking for examples of Partially observed Markov decision processes (preferably infinite horizon, Discrete time, Discrete ...
0
votes
1answer
145 views

Supremum in a Markov chain model

A Markov chain $X$ with finite state space $\{1,2,\cdots,N\}$ is defined on a probability space $(\Omega, P, \mathcal{F})$ equiped with filtration $\{\mathcal{F}_t\}$. And we assume that we can reach ...
1
vote
1answer
98 views

Kalman filter with long term bias

I was reading about the Kalman filter and I do not understand how it should be used when our measurements have a long term offset like GPS location updates do. As I understand, the Kalman filter ...
0
votes
1answer
253 views

Anyone has Kushner's book “Introduction to stochastic control” 1971? I need a theorem from it

In a paper I'm reading, it refers to Theorem 8, Page 217 of the book "Introduction to Stochastic Control" H. J. Kushner, New York: Holt, Reinhart, and Winston 1971. Unfortunately I don't have it and ...
2
votes
1answer
83 views

Maximal probability of “infinitely often” over MDP

Let us consider a Markov Decision Process (MDP) with a Borel state space $X$. Often, the optimization problems over MDP involve optimization of some objectives dependent on the reward function $$ ...
4
votes
0answers
69 views

Sufficiency of stationary policy for negative stochastic dynamic programming

Consider a Markov Decision Process with Borel state space $X$ and Borel action space $U$, like the one defined in the book "Stochastic Optimal Control: Discrete-time case" by Bertsekas and Shreve. All ...
2
votes
1answer
136 views

Upper bound concerning Snell envelope

Consider, on a filtred probability space $ \left (\Omega, \mathcal F, \mathbb F , \mathbb P \right )$ where $ \mathbb F = \left(\mathcal F_ t \right )_ {t\geq 0}$ is filtration satisfying the usuual ...
0
votes
0answers
144 views

Stochastic optimal control with no diffusion

Classical stochastic optimal control problem is to minimize functional $$ J(u) = \mathbf E \int_0^T f(t,x_t,u_t)dt, \tag{1} $$ subject to SDE $$ dx_t = b(t,x_t,u_t)dt + \sigma(t,x_t,u_t)dW_t, \quad ...
3
votes
0answers
147 views

stochastic control / geometric mean

Consider the following problem: Given $\Omega$ and $U$ two symmetric definite positive matrices, choose a matrix $K$ to minimize the expectation $x' \Omega x + x'K'UKx$ when $x$ follows the invariant ...
1
vote
3answers
433 views

convergence in distribution of stochastic gradient descent.

The stochastic gradient descent algorithm where only a noisy gradient (zero mean noise) is used to update current estimate is known to converge almost surely to the minimizer. However, if one is ...
4
votes
2answers
1k views

Commuting supremum and expectation

Given a one-parametric random function on a probability space $(\Omega,\mathcal F,\mathbb P)$: $X:U\times\Omega\to \mathbb R \text{ and } (a,w)\mapsto X(a,w), \text{ with } \sigma(X(a))\subseteq ...