All Questions
Tagged with estimation-theory st.statistics
44 questions
1
vote
3
answers
162
views
Upper Bound for $\mathbb{E}\left[\max_{j \in \mathcal{N}} h_{j}\right]$
Assume $\{h_j\}_{j\in \mathcal{N}}$ are independent Gamma random variables, each with potentially different distributions and parameters. I am looking for an upper bound for $\mathbb{E}\left[\max_{j \...
- 41
1
vote
1
answer
56
views
How to study the convergence of the sample mode for arbitrary probability spaces
(This is not the problem I actually care about, but an analogy with similar issues to the problem I'm actually considering.)
Consider a probability space with i.i.d. random variables $X_i$ producing ...
- 277
2
votes
1
answer
170
views
Equivalence of minimizing trace and determinant over matrix quadratic form in multivariate regression
Consider the multivariate regression model
$$Y = XB + E$$
where $Y$ is $n \times p$ and corresponds to the dependent variables, $X$ is $n \times k$ and corresponds to the independent variables, $B$ is ...
1
vote
0
answers
148
views
conjecture for general form of minimax estimator
I had previously posed an overly ambitious version of this conjecture here,
Form of minimax estimator,
which was quickly shot down by Václav Voráček (on twitter) and Iosif Pinelis (MO answer in the ...
- 6,541
1
vote
1
answer
341
views
Form of minimax estimator
Let $\Delta$ be the set of all probability distributions over $\mathbb{N}=\{1,2,\ldots\}$ and fix some $\mathcal{P}\subseteq\Delta$.
Suppose additionally that $\Delta$ is endowed with some norm $||\...
- 6,541
0
votes
1
answer
940
views
Derivative of log-likelihood function for Gaussian distribution with parameterized variance
Suppose we have a parameter $\theta \in R^{n}$ that defines some noisy observation $z=\mu(\theta)+\eta, z\in R^{m}$ where the noise follows a Gaussian distribution whose covariance is a function of ...
- 75
7
votes
1
answer
569
views
Singular Fisher information matrix and existence of unbiased estimators
I'm doing some research into the Cramer-Rao bound for time of arrival localization and have come across a rather strange result: the FIM is singular, but there exists an unbiased estimator. My ...
- 75
2
votes
0
answers
87
views
A complex problem involving densities (likelihood functions) and optimization
Consider the following autoregressive process with normal errors:
\begin{equation}\label{7YlUV4i8nuO}\tag{I}
y_t = \phi y_{t-1}+ u_t, \quad u_t \overset{iid}{\sim} N(0,\sigma^2)
\end{equation}
We ...
- 13
2
votes
1
answer
143
views
DKW inequality for $L^1$-norm
Suppose that $X,X_1,X_2,X_3\dots$ is a sequence of $\mathbb{P}$-i.i.d. random variables supported in the interval $[0,1]$. Let $F$ be the cumulative distribution of $X$, i.e. $F(x):=\mathbb{P}[X \le x]...
- 123
1
vote
0
answers
34
views
Correlating two matrices $A,B$ with stochastic dependency structure imposed by cross-validation
Consider a labelled data set
$$D = \{(x_1, y_1),...,(x_n, y_n)\} $$
on which we want to evaluate a machine learning algorithm using $k$-fold cross validation with $m$ different random seeds. This ...
- 153
4
votes
1
answer
415
views
What journal(s) do you recommend for submitting a paper on a topic that spans information theory and estimation theory?
I've written a paper that a) demonstrates an equivalence between conditional complexity $K$($Y$|$X$) in information theory and the random component of an effect size estimate $r_{xy}$, and then b) ...
- 183
2
votes
0
answers
78
views
Distribution of unbiased estimator of covariance matrix with missing values
Initial setup
Assuming $X_1, ..., X_n \in \mathbb{R}^m$ are iid, sampled from $\mathcal{N}(\mu, V)$, one can define the estimators for the sample mean $\hat{\mu} = \frac{1}{n} := X^T 1_n$, and sample ...
- 255
2
votes
0
answers
130
views
L1 error of estimators
I came across the following problem and I have no clue how to approach it. I am looking for help with directions or references.
Consider the $\alpha$-stable distribution with unknown true mean $\mu$, ...
- 173
1
vote
0
answers
75
views
Percentile interval Lemma
Let $\theta$ be a parameter and $\hat{\theta}$ the plug-in estimate, I need a proof of the following lemma, as given in [1], p. 173, in the form of a reference or of a direct argument:
Percentile ...
1
vote
0
answers
108
views
Bootstrap-$t$ confidence intervals
I'm writing a dissertation about bootstrap methods and the main book I'm using is Efron, B., & Tibshirani, R.J. (1994), An Introduction to the Bootstrap (1st ed.), Chapman and Hall/CRC. Now I need ...
3
votes
0
answers
113
views
Image restoration quality general lower bounds
A typical image restoration model posits that, starting from a true image $f = f(x,y)$, we observe
$$
\tilde f = f \star h + n
$$
where $\star$ is convolution, $h$ is the point spread function (caused,...
- 1,069
1
vote
1
answer
170
views
Error metric for joint estimation of mean and variance
Background:
Let $\mu:\mathbb{R}^n\to\mathbb{R}$ and $\sigma:\mathbb{R}^n\to\mathbb{R}_+$ be two unknown functions, and consider a stochastic model of the form
$$
\mathbb{E}[Y\mid\mathbf{x}] = \mu(\...
- 617
-2
votes
1
answer
92
views
Existence or impossibility of Gaussian factory
Gaussian factory problem: given an iid sequence $x_i \sim \mathcal{N}(\mu,\sigma^2)$, $i=1,2,\dots$, with $\mu$ and $\sigma^2$ both unknown, construct a realization $y \sim \mathcal{N}(0,1)$.
3
votes
1
answer
87
views
Optimal linear measurement operator
Let $x\in R^n$ be an unknown vector. Suppose I am allowed to choose any $A\in R^{m\times n}$, under the constraint that each row of $A$ has $\ell_2$ norm at most $1$. Then I carry out a "measurement", ...
- 6,541
2
votes
1
answer
676
views
Distribution of ratio between complex Gaussian and Chi-square R.V.s
What would be the distribution (p.d.f.) of the following ratio?
$$z = \frac{x_{1}}{|x_{1}|^2 + |x_{2}|^2 + ... + |x_{M}|^2}$$
where $x_{i} \sim \mathcal{CN}(0,a), \forall i$ and $a > 1$. As can ...
2
votes
2
answers
632
views
An alternative proof of Bayesian Cramer-Rao
My question is:
Are there an alternative proof of Cramer-Rao lower bound that does not use
Cauchy-Swartz inequality?
Let me outline the classical proof and explain why I am interested in this ...
- 671
2
votes
0
answers
56
views
Rate of $L_1$ loss in estmating density on $[0,1]$
Let $f$ be a density on $[0,1]$ and let $X_1,X_2,\ldots$ be $\textit{iid}$ $f$-distributed. Also, let $f_n$ denote the kernel density estimator, i.e.
$$f_n(x) = \frac{1}{nh_n} \sum_{i=1}^n K\left(\...
- 121
1
vote
1
answer
193
views
Fisher information with vanishing probability
I am confused about the definition of the Fisher information and the case when probability is 0. Consider discrete set $\epsilon$ of possible measurement outcomes. Fisher information is defined as:
$$...
- 237
1
vote
0
answers
93
views
A different objective function in liner regression analysis
I'm an undergraduate student who is green in statistics. I have a problem in the chose of objective function when estimating the parameters.
Let $Y = \beta^TX + \epsilon $ be the standard liner ...
- 11
1
vote
0
answers
79
views
sufficient statistics that are irrelevant
I'm designing a lecture on hypothesis testing and want to do an example on a certain matter, but I cannot come up with a good one.
If we should decide upon $H_0$ or $H_1$ given observed data sets ${\...
- 129
2
votes
0
answers
119
views
Calculate sample mean confidence interval of noisy logistical distribution
I have $n$ samples which follow a logistic distribution with unknown $u$ and $s$; it is affected by a Gaussian noise with 0 mean.
I would like to estimate its average $u$ with a confidence interval (...
- 21
3
votes
1
answer
96
views
Perturbation results for statistical estimators
Suppose I have a continuous random variable whose distribution $f$ is some parametric form (normal, exponential, etc.) that is known to me. If I draw many independent samples $x_i$ from $f$, I can ...
- 4,049
3
votes
0
answers
82
views
Uniform mean-square-error estimates
Consider a standard statistical estimation problem with iid real observations $\{X_i\}_{i=1}^N$. For a collection of real functions $\mathcal{F}$, I want to get an estimate of the uniform rate of ...
- 111
4
votes
1
answer
203
views
Can samples be compressed?
The Fisher information of a random variable $Y$ about a parameter $\theta$ upon which the probability of $Y$ depends is:
$\mathcal{I}_Y(\theta)= -E\left[\left.\strut \frac{\partial^2}{\partial \theta^...
- 22.1k
2
votes
0
answers
72
views
Robust weighted estimator of location
Let $X = (x_1, \ldots, x_n)$ be a sample of i.i.d values. There are several robust estimators of sample location, most notably sample median and Hodges-Lehmann estimator.
Now let $W = (w_1, \ldots, ...
- 317
1
vote
1
answer
256
views
KL divergence Inequality
I am trying to find a proof for the following inequality, but I did not get anywhere following the references from the paper I was reading.
Consider two probability measures $P$ and $Q$ both ...
- 275
4
votes
2
answers
519
views
Cramér–Rao type bound for absolute estimation error
Let $\{X_1, X_2, \dotsc, X_n\}$ be independent and identically distributed (i.i.d.) random variables sampled from a common distribution with density $f_{\theta}(x)$, where $\theta$ is an unknown ...
- 544
2
votes
1
answer
88
views
What is the problem with this model parameter estimation algorithm?
In a statistical model with parameters $\theta$ and unobserved laten variables $Z$, the model likelihood is
$$L(\theta;X)=Pr(X|\theta)=\sum_ZPr(X,Z|\theta)$$
The standard way to estimate $\theta$ ...
- 65
2
votes
3
answers
409
views
How to estimate the entropy of a distribution on a power set?
Given a probability distribution $(X,p)$, its entropy is defined as $H=-\sum_{x\in X} p(x)\log p(x)$.
Given a sample of observations $x_n,n=1..N$, one can estimate $p(x)=\frac{\#\{i:x_i=x\}}{N}$ and ...
- 165
2
votes
2
answers
174
views
estimating variance of dependent normal distributed data
Let $X_{ij}$ with $1\leq i<j\leq n$ (that are $X_{12},\dots, X_{1n},\dots,X_{(n-1)n}$) be ${n \choose 2}$ identically normal distributed $N(0,\sigma^2)$ such that
$
\text{corr}(X_{ij},X_{rs})=\rho
...
- 133
4
votes
1
answer
288
views
Equivalent method for maximum likelihood estimation of covariance parameters
My goal is to estimate the parameters of a covariance matrix $\Omega$, by maximizing the following log-likelihood function:
$$\log L(\vec\tau, \rho, \sigma \mid W, X) = -m\ln(\left | \Omega \right |) ...
- 141
1
vote
0
answers
186
views
Shrinkage (or Stein's phenomenon) in low dimensions, discrete contexts
I am trying to understand shrinkage, or the Stein phenomenon. As someone without a statistics background, the focus in most introductory presentations on normal distributions and squared error loss ...
- 203
5
votes
0
answers
190
views
Pair of two-variable polynomial equations of high order
I have the following pair of equations to be solved for two variables $\rho$ and $D$ resulting from a certain Maximum Likelihood Estimation for a time series $X_n > 0$, $n=0, \ldots, N+1$ with $N \...
- 548
1
vote
1
answer
282
views
Is an unbiased estimator with arbitrarily small variance necessarily consistent?
Given an unbiased estimator $\hat \theta_n$ of a parameter $\theta$, if the estimator has small variance (approaching $0$ as $n\to\infty$), it seems reasonable to expect that the estimator is ...
- 133
0
votes
0
answers
185
views
Why does the OLS estimator simplify as follows for the single regressor case?
I was reading in "A Guide to Econometrics" that given $Y = X \beta + \epsilon$, the variance covariance matrix of $\beta^\text{OLS}$ is given by $\sigma^2 (X' X)^{-1}$ where $\sigma^2$ is the variance ...
- 1
3
votes
1
answer
578
views
Why doesn't Stein effect happen for multinomial distributions?
(Medeen, et all, 1998)" show that Maximum Likelihood estimate is admissible for multinomial distribution under squared error. On other hand, James and Stein showed that arithmetic average is not an ...
- 2,252
20
votes
1
answer
4k
views
Using Fisher Information to bound KL divergence
Is it possible to use Fisher Information at p to get a useful upper bound on KL(q,p)?
KL(q,p) is known as Kullback-Liebler divergence and is defined for discrete distributions over k outcomes as ...
- 2,252
1
vote
1
answer
313
views
Is there a text on estimation theory online?
Where can I find graduate level, thorough, parameter estimation/ estimation theory material on the web?
- 307
3
votes
3
answers
2k
views
Is the min function ever an unbiased estimator for the mean?
Given $n$ i.i.d. variables $X_1$ to $X_n$ with an unknown probability distribution, the sample average is an unbiased estimator for the mean of the distribution. Is there some non-trivial probability ...
- 597