All Questions
23 questions
20
votes
1
answer
4k
views
Using Fisher Information to bound KL divergence
Is it possible to use Fisher Information at p to get a useful upper bound on KL(q,p)?
KL(q,p) is known as Kullback-Liebler divergence and is defined for discrete distributions over k outcomes as ...
- 2,252
7
votes
1
answer
569
views
Singular Fisher information matrix and existence of unbiased estimators
I'm doing some research into the Cramer-Rao bound for time of arrival localization and have come across a rather strange result: the FIM is singular, but there exists an unbiased estimator. My ...
- 75
4
votes
1
answer
203
views
Can samples be compressed?
The Fisher information of a random variable $Y$ about a parameter $\theta$ upon which the probability of $Y$ depends is:
$\mathcal{I}_Y(\theta)= -E\left[\left.\strut \frac{\partial^2}{\partial \theta^...
- 22.1k
4
votes
2
answers
519
views
Cramér–Rao type bound for absolute estimation error
Let $\{X_1, X_2, \dotsc, X_n\}$ be independent and identically distributed (i.i.d.) random variables sampled from a common distribution with density $f_{\theta}(x)$, where $\theta$ is an unknown ...
- 544
4
votes
1
answer
288
views
Equivalent method for maximum likelihood estimation of covariance parameters
My goal is to estimate the parameters of a covariance matrix $\Omega$, by maximizing the following log-likelihood function:
$$\log L(\vec\tau, \rho, \sigma \mid W, X) = -m\ln(\left | \Omega \right |) ...
- 141
3
votes
3
answers
2k
views
Is the min function ever an unbiased estimator for the mean?
Given $n$ i.i.d. variables $X_1$ to $X_n$ with an unknown probability distribution, the sample average is an unbiased estimator for the mean of the distribution. Is there some non-trivial probability ...
- 597
3
votes
1
answer
578
views
Why doesn't Stein effect happen for multinomial distributions?
(Medeen, et all, 1998)" show that Maximum Likelihood estimate is admissible for multinomial distribution under squared error. On other hand, James and Stein showed that arithmetic average is not an ...
- 2,252
3
votes
1
answer
87
views
Optimal linear measurement operator
Let $x\in R^n$ be an unknown vector. Suppose I am allowed to choose any $A\in R^{m\times n}$, under the constraint that each row of $A$ has $\ell_2$ norm at most $1$. Then I carry out a "measurement", ...
- 6,541
3
votes
1
answer
96
views
Perturbation results for statistical estimators
Suppose I have a continuous random variable whose distribution $f$ is some parametric form (normal, exponential, etc.) that is known to me. If I draw many independent samples $x_i$ from $f$, I can ...
- 4,049
3
votes
0
answers
82
views
Uniform mean-square-error estimates
Consider a standard statistical estimation problem with iid real observations $\{X_i\}_{i=1}^N$. For a collection of real functions $\mathcal{F}$, I want to get an estimate of the uniform rate of ...
- 111
2
votes
2
answers
632
views
An alternative proof of Bayesian Cramer-Rao
My question is:
Are there an alternative proof of Cramer-Rao lower bound that does not use
Cauchy-Swartz inequality?
Let me outline the classical proof and explain why I am interested in this ...
- 671
2
votes
1
answer
676
views
Distribution of ratio between complex Gaussian and Chi-square R.V.s
What would be the distribution (p.d.f.) of the following ratio?
$$z = \frac{x_{1}}{|x_{1}|^2 + |x_{2}|^2 + ... + |x_{M}|^2}$$
where $x_{i} \sim \mathcal{CN}(0,a), \forall i$ and $a > 1$. As can ...
2
votes
0
answers
87
views
A complex problem involving densities (likelihood functions) and optimization
Consider the following autoregressive process with normal errors:
\begin{equation}\label{7YlUV4i8nuO}\tag{I}
y_t = \phi y_{t-1}+ u_t, \quad u_t \overset{iid}{\sim} N(0,\sigma^2)
\end{equation}
We ...
- 13
1
vote
1
answer
341
views
Form of minimax estimator
Let $\Delta$ be the set of all probability distributions over $\mathbb{N}=\{1,2,\ldots\}$ and fix some $\mathcal{P}\subseteq\Delta$.
Suppose additionally that $\Delta$ is endowed with some norm $||\...
- 6,541
1
vote
1
answer
282
views
Is an unbiased estimator with arbitrarily small variance necessarily consistent?
Given an unbiased estimator $\hat \theta_n$ of a parameter $\theta$, if the estimator has small variance (approaching $0$ as $n\to\infty$), it seems reasonable to expect that the estimator is ...
- 133
1
vote
1
answer
56
views
How to study the convergence of the sample mode for arbitrary probability spaces
(This is not the problem I actually care about, but an analogy with similar issues to the problem I'm actually considering.)
Consider a probability space with i.i.d. random variables $X_i$ producing ...
- 277
1
vote
1
answer
256
views
KL divergence Inequality
I am trying to find a proof for the following inequality, but I did not get anywhere following the references from the paper I was reading.
Consider two probability measures $P$ and $Q$ both ...
- 275
1
vote
1
answer
193
views
Fisher information with vanishing probability
I am confused about the definition of the Fisher information and the case when probability is 0. Consider discrete set $\epsilon$ of possible measurement outcomes. Fisher information is defined as:
$$...
- 237
1
vote
0
answers
148
views
conjecture for general form of minimax estimator
I had previously posed an overly ambitious version of this conjecture here,
Form of minimax estimator,
which was quickly shot down by Václav Voráček (on twitter) and Iosif Pinelis (MO answer in the ...
- 6,541
1
vote
0
answers
75
views
Percentile interval Lemma
Let $\theta$ be a parameter and $\hat{\theta}$ the plug-in estimate, I need a proof of the following lemma, as given in [1], p. 173, in the form of a reference or of a direct argument:
Percentile ...
1
vote
0
answers
186
views
Shrinkage (or Stein's phenomenon) in low dimensions, discrete contexts
I am trying to understand shrinkage, or the Stein phenomenon. As someone without a statistics background, the focus in most introductory presentations on normal distributions and squared error loss ...
- 203
0
votes
1
answer
940
views
Derivative of log-likelihood function for Gaussian distribution with parameterized variance
Suppose we have a parameter $\theta \in R^{n}$ that defines some noisy observation $z=\mu(\theta)+\eta, z\in R^{m}$ where the noise follows a Gaussian distribution whose covariance is a function of ...
- 75
-2
votes
1
answer
92
views
Existence or impossibility of Gaussian factory
Gaussian factory problem: given an iid sequence $x_i \sim \mathcal{N}(\mu,\sigma^2)$, $i=1,2,\dots$, with $\mu$ and $\sigma^2$ both unknown, construct a realization $y \sim \mathcal{N}(0,1)$.
