Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [estimation-theory]

The tag has no usage guidance.

5
votes
1answer
58 views

Estimating the size of the remainder in a random partition

Pick a sequence of real numbers $x_i$ as follows. Put $x_0=1$. If $x_i$ is chosen, then pick $x_{i+1}\in[0, x_i]$ according to the uniform distribution. Obviously we have $x_i\rightarrow 0$ with ...
-2
votes
1answer
74 views

Existence or impossibility of Gaussian factory

Gaussian factory problem: given an iid sequence $x_i \sim \mathcal{N}(\mu,\sigma^2)$, $i=1,2,\dots$, with $\mu$ and $\sigma^2$ both unknown, construct a realization $y \sim \mathcal{N}(0,1)$.
0
votes
0answers
60 views

Nonparametric Bayesian estimation of several black-box functions of different variables from their noisy sums

In order to introduce my problem, let’s start with the nonparametric estimation of a single unknown/black-box function $f:{\Omega _f} \to \mathbb{R}$ of a discrete variable $x$ in a finite domain ${\...
0
votes
1answer
45 views

Strict positive type function on hypersurface also of positive type in neighborhood?

Let $u\in C^\infty(\mathbb{R}^n\times\mathbb{R}^n)$ be symmetric and of strictly positive type on some hypersurface $S \subset \mathbb{R}^n$ diffeomorphic to $\{0\}\times\mathbb{R}^{n-1}$. This means ...
0
votes
0answers
26 views

Efficiency of importance sampling in terms of the size of the the support of sampling distribution

In importance sampling, one proposes to compute an integral $I:=\mathbb E_{x \sim P}[h(x)]$ by rewritting it as $$ I=\mathbb E_{x \sim Q}\left[w(x)h(x)\right],\text{ with }w(x):=\frac{p(x)}{q(x)}, $$ ...
3
votes
1answer
70 views

Optimal linear measurement operator

Let $x\in R^n$ be an unknown vector. Suppose I am allowed to choose any $A\in R^{m\times n}$, under the constraint that each row of $A$ has $\ell_2$ norm at most $1$. Then I carry out a "measurement", ...
1
vote
0answers
142 views

Functional Taylor expansion for differential entropy

Consider an continuous distribution $F$ with density $f$. The (differential) Shannon entropy of $f$ is $h(f)=-\int f(x)\log f(x) dx$. In the literature of differential entropy estimation, ...
2
votes
1answer
141 views

Finding a similarities and differences of sent of matrices

Suppose we have a set of rank deficient covariance matrices. How can I know the similarities and differences between those set of matrices? Regards,
2
votes
1answer
171 views

Distribution of ratio between complex Gaussian and Chi-square R.V.s

What would be the distribution (p.d.f.) of the following ratio? $z = \frac{x_{1}}{|x_{1}|^2 + |x_{2}|^2 + ... + |x_{M}|^2}$ where $x_{i} \sim \mathcal{CN}(0,a), \forall i$ and $a > 1$. As can be ...
3
votes
1answer
122 views

Proving bounds on analytic functions using only the Taylor expansion

I wonder if there is a general method for obtaining bounds on an analytic function using only its Taylor expansion (not using its special properties such as satisfying a good differential equation, ...
3
votes
1answer
99 views

maximum likelihood estimation of X is better than that of f(X)?

Consider a binary variable $C$ with $\Pr(C=0)=\Pr(C=1)=0.5$. Consider a random vector $X \in \mathbb{R}^d$, characterized by $C$, with PDF $p_m(x)$, $m\in\{0,1\}$. Define a maximum likelihood (ML) ...
2
votes
2answers
273 views

An alternative proof of Bayesian Cramer-Rao

My question is: Are there an alternative proof of Cramer-Rao lower bound that does not use Cauchy-Swartz inequality? Let me outline the classical proof and explain why I am interested in this ...
1
vote
0answers
67 views

Extended Kalman filter for initial values estimation

I try to make extended Kalman filter for estimation of initial values of small celestial body. I have: $(x_1^0, x_2^0, x_3^0, v_1^0, v_2^0, v_3^0) = (x^0, v^0)$ -- inaccurate initial values. $z = ...
2
votes
0answers
45 views

Rate of $L_1$ loss in estmating density on $[0,1]$

Let $f$ be a density on $[0,1]$ and let $X_1,X_2,\ldots$ be $\textit{iid}$ $f$-distributed. Also, let $f_n$ denote the kernel density estimator, i.e. $$f_n(x) = \frac{1}{nh_n} \sum_{i=1}^n K\left(\...
1
vote
1answer
93 views

Fisher information with vanishing probability

I am confused about the definition of the Fisher information and the case when probability is 0. Consider discrete set $\epsilon$ of possible measurement outcomes. Fisher information is defined as: $$...
1
vote
0answers
78 views

A different objective function in liner regression analysis

I'm an undergraduate student who is green in statistics. I have a problem in the chose of objective function when estimating the parameters. Let $Y = \beta^TX + \epsilon $ be the standard liner ...
1
vote
1answer
113 views

Reconstructing the number of distinct elements from a random projection

Assume we have an unknown sequence $x_1,\ldots, x_n\in \mathcal U$. We get to observe the sequence $h(x_1),h(x_2),\ldots, h(x_n)$, where $h:\mathcal U\to \{1,\ldots, k\}$ is a random function such ...
1
vote
0answers
70 views

sufficient statistics that are irrelevant

I'm designing a lecture on hypothesis testing and want to do an example on a certain matter, but I cannot come up with a good one. If we should decide upon $H_0$ or $H_1$ given observed data sets ${\...
1
vote
0answers
43 views

A question about the prediction error

I am reading about the prediction error estimation and I found the following: Suppose we have ${\mathbf{Y}}=\mathbf{x}_0+ \epsilon$, where, $\epsilon$ is normally distributed as $\sim \mathcal{N}(0, \...
2
votes
0answers
40 views

Calculate sample mean confidence interval of noisy logistical distribution

I have $n$ samples which follow a logistic distribution with unknown $u$ and $s$; it is affected by a Gaussian noise with 0 mean. I would like to estimate its average $u$ with a confidence interval (...
1
vote
0answers
52 views

Cramer Rao bound for relative estimation

I have an observed vector ${\bf y}$ from which I would like to estimate a parameter vector ${\bf c}$ (denote the estimate $\hat{{\bf c}}$). A feature of our estimation problem is that the involved ...
2
votes
0answers
38 views

Perturbation results for statistical estimators

Suppose I have a continuous random variable whose distribution $f$ is some parametric form (normal, exponential, etc.) that is known to me. If I draw many independent samples $x_i$ from $f$, I can ...
2
votes
0answers
70 views

Uniform mean-square-error estimates

Consider a standard statistical estimation problem with iid real observations $\{X_i\}_{i=1}^N$. For a collection of real functions $\mathcal{F}$, I want to get an estimate of the uniform rate of ...
5
votes
1answer
140 views

Can samples be compressed?

The Fisher information of a random variable $Y$ about a parameter $\theta$ upon which the probability of $Y$ depends is: $\mathcal{I}_Y(\theta)= -E\left[\left.\strut \frac{\partial^2}{\partial \theta^...
1
vote
0answers
54 views

Robust weighted estimator of location

Let $X = (x_1, \ldots, x_n)$ be a sample of i.i.d values. There are several robust estimators of sample location, most notably sample median and Hodges-Lehmann estimator. Now let $W = (w_1, \ldots, ...
1
vote
0answers
34 views

Bounding Hidden Markov model Bayesian filter error with inexact models

In context of a hidden Markov model, I am interested in bounding the error of a Bayesian filter when using inexact state transition and observation models. Consider a hidden Markov model (HMM) with ...
2
votes
0answers
62 views

Elliptic Equation with Wentzell boundary condition

I'm looking for a reference showing how to obtain a priori estimate for solutions to a linear second-order elliptic equation with Wentzell boundary condition in a bounded domain in $H^1$ space. The ...
2
votes
1answer
144 views

Literature question on the convergence rate of the empirical distribution

Assume that given $n$ i.i.d samples $(X_1, X_2, ..., X_n)$ drawn from $p_X$, an unknown probability mass function defined over a finite alphabet $\mathcal{X}$, one wants to estimate $p_X(x)$ for each $...
2
votes
0answers
135 views

Maximum-likelihood estimation for univariate responses from multivariate data

I am new in the field of machine learning, so I hope I will be able to formulate my question in a clear way... I have some data represented by vectors $\mathbf{x}_1,\mathbf{x}_2,\ldots,\mathbf{x}_n \...
1
vote
1answer
209 views

KL divergence Inequality

I am trying to find a proof for the following inequality, but I did not get anywhere following the references from the paper I was reading. Consider two probability measures $P$ and $Q$ both ...
2
votes
0answers
119 views

How to find moment condition for generalized method of moments?

Consider a scalar system with $2K$ outputs and $K+2$ unknowns: $y_{k,1}=x_ka_1+n_{k,1} \quad y_{k,2}=x_ka_2+n_{k,1}$. The variables $n_{k,\ell}$ are zero mean noise variables. To estimate $a_1$ and $...
4
votes
1answer
169 views

Cramér-Rao bound for randomized estimator

As is well known, the Cramér-Rao bound (or information inequality) sets a lower bound on the variance of estimators of a parameter. Consider the case when the parameter is a scalar, the estimator is ...
2
votes
0answers
45 views

MLE and CRLB with mismatched likelihoods

Suppose that I can do a Karhunen-Loeve expansion of a log-likelihood function $p(\bf{x};\theta)$ into N terms and that these accounts for a fraction $1-\delta$ of the total energy. Now consider ...
4
votes
1answer
290 views

Cramer-Rao type bound for absolute estimation error

Let $\{X_1, X_2, \ldots, X_n\}$ be independent and identically distributed (i.i.d.) random variables sampled from a common distribution with density $f_{\theta}(x)$, where $\theta$ is an unknown ...
1
vote
0answers
41 views

Equivalence of Graphical model selection algorithms

Suppose, a jointly Gaussian random vector is denoted by $X \in \mathbb{R}^{p}$ and $X$ has a distribution given by $\mathcal{N}(\mu,\Sigma)$. It is known that estimating the graphical model that ...
1
vote
0answers
210 views

Distribution of a signal covariance matrix

A common estimation problem in signal processing assumes the following signal model \begin{equation} \mathbf{r} = \sum_{i=1}^{Q}\alpha_i\mathbf{s}\left(w_i\right)+\mathbf{n} \end{equation} where $\...
2
votes
1answer
82 views

Estimating mean and variance of a distribution based on error-prone estimates of its cdf

Suppose I have some random variable $X$ taking values in $[a, b]$ with unknown distribution (I am happy to assume the distribution is smooth, though it would be nice to not have to). I have a ...
2
votes
1answer
79 views

What is the problem with this model parameter estimation algorithm?

In a statistical model with parameters $\theta$ and unobserved laten variables $Z$, the model likelihood is $$L(\theta;X)=Pr(X|\theta)=\sum_ZPr(X,Z|\theta)$$ The standard way to estimate $\theta$ ...
2
votes
3answers
222 views

How to estimate the entropy of a distribution on a power set?

Given a probability distribution $(X,p)$, its entropy is defined as $H=-\sum_{x\in X} p(x)\log p(x)$. Given a sample of observations $x_n,n=1..N$, one can estimate $p(x)=\frac{\#\{i:x_i=x\}}{N}$ and ...
2
votes
2answers
169 views

estimating variance of dependent normal distributed data

Let $X_{ij}$ with $1\leq i<j\leq n$ (that are $X_{12},\dots, X_{1n},\dots,X_{(n-1)n}$) be ${n \choose 2}$ identically normal distributed $N(0,\sigma^2)$ such that $ \text{corr}(X_{ij},X_{rs})=\rho ...
3
votes
2answers
115 views

Markov-type functions

I'd like to have some informations about Markov-type functions (or Cauchy-type): \[ f(z)=\int_{\Gamma} \frac{\mathrm{d}\gamma(\xi)}{\xi-z}.\] $\gamma$ is a positive measure with compact support $\...
1
vote
0answers
72 views

Stochastic process inference from partial observations

Consider a set $U$. My signal is a piece-wise constant "function" $Sig: t \mapsto s$, i.e. the signal at time $t$ equals to some subset $s \subset U$. One can see $Sig(t)$ as a stochastic process. ...
3
votes
1answer
220 views

Equivalent method for maximum likelihood estimation of covariance parameters

My goal is to estimate the parameters of a covariance matrix $\Omega$, by maximizing the following log-likelihood function: $$\log L(\vec\tau, \rho, \sigma \mid W, X) = -m\ln(\left | \Omega \right |) ...
0
votes
1answer
74 views

About the suboptimality of linear estimators

Let $X$ be a random variable and $N$ a Gaussian noise independent from $X$. We observe $Y=X+N$ and want to estimate $X$ based on $Y$ to minimize the mean square error $mmse(X|Y):=E(\hat X(Y)-X)^2$. ...
3
votes
2answers
375 views

Moments of Matrix Gamma distribution

Matrix gamma distribution (defined for example in http://en.wikipedia.org/wiki/Matrix_gamma_distribution) is one way to generalize Wishart distribution. In our course work that distribution was used ...
1
vote
0answers
147 views

Shrinkage (or Stein's phenomenon) in low dimensions, discrete contexts

I am trying to understand shrinkage, or the Stein phenomenon. As someone without a statistics background, the focus in most introductory presentations on normal distributions and squared error loss ...
4
votes
0answers
177 views

Pair of two-variable polynomial equations of high order

I have the following pair of equations to be solved for two variables $\rho$ and $D$ resulting from a certain Maximum Likelihood Estimation for a time series $X_n > 0$, $n=0, \ldots, N+1$ with $N \...
0
votes
1answer
109 views

Signal model classification between two possbile candidates

How to decide the most possible signal model between two model candidates besed on the received signal vector? Assume the received signal vector is $y$, the possible signal model candidates could be: ...
2
votes
0answers
202 views

Worst-case error and Cramer-Rao Lower Bound - is there any mathematical relation between them?

I would like to understand the relation (if any) between the Cramer-Rao Lower Bound of estimation theory and the following simple definition of "reconstruction accuracy" which doesn't use any ...
2
votes
0answers
789 views

Definition and Convergence of Iteratively Reweighted Least Squares

I've been using iteratively reweighted least squares (IRLS) to minimize functions of the following form, $J(m) = \sum_{i=1}^{N} \rho \left(\left| x_i - m \right|\right)$ where $N$ is the number of ...