All Questions
Tagged with estimation-theory it.information-theory
9 questions
20
votes
1
answer
4k
views
Using Fisher Information to bound KL divergence
Is it possible to use Fisher Information at p to get a useful upper bound on KL(q,p)?
KL(q,p) is known as Kullback-Liebler divergence and is defined for discrete distributions over k outcomes as ...
- 2,252
4
votes
1
answer
203
views
Can samples be compressed?
The Fisher information of a random variable $Y$ about a parameter $\theta$ upon which the probability of $Y$ depends is:
$\mathcal{I}_Y(\theta)= -E\left[\left.\strut \frac{\partial^2}{\partial \theta^...
- 22.1k
4
votes
1
answer
415
views
What journal(s) do you recommend for submitting a paper on a topic that spans information theory and estimation theory?
I've written a paper that a) demonstrates an equivalence between conditional complexity $K$($Y$|$X$) in information theory and the random component of an effect size estimate $r_{xy}$, and then b) ...
- 183
2
votes
3
answers
409
views
How to estimate the entropy of a distribution on a power set?
Given a probability distribution $(X,p)$, its entropy is defined as $H=-\sum_{x\in X} p(x)\log p(x)$.
Given a sample of observations $x_n,n=1..N$, one can estimate $p(x)=\frac{\#\{i:x_i=x\}}{N}$ and ...
- 165
2
votes
0
answers
51
views
MLE and CRLB with mismatched likelihoods
Suppose that I can do a Karhunen-Loeve expansion of a log-likelihood function $p(\bf{x};\theta)$ into N terms and that these accounts for a fraction $1-\delta$ of the total energy. Now consider ...
- 21
1
vote
0
answers
34
views
Correlating two matrices $A,B$ with stochastic dependency structure imposed by cross-validation
Consider a labelled data set
$$D = \{(x_1, y_1),...,(x_n, y_n)\} $$
on which we want to evaluate a machine learning algorithm using $k$-fold cross validation with $m$ different random seeds. This ...
- 153
1
vote
0
answers
79
views
sufficient statistics that are irrelevant
I'm designing a lecture on hypothesis testing and want to do an example on a certain matter, but I cannot come up with a good one.
If we should decide upon $H_0$ or $H_1$ given observed data sets ${\...
- 129
1
vote
0
answers
62
views
Cramer Rao bound for relative estimation
I have an observed vector ${\bf y}$ from which I would like to estimate a parameter vector ${\bf c}$ (denote the estimate $\hat{{\bf c}}$).
A feature of our estimation problem is that the involved ...
- 21
0
votes
0
answers
31
views
What is the Fisher information matrix of the von Mises-Fisher distribution?
Assuming the von Mises-Fisher distribution as
$$f_{p}(\mathbf{x}; \boldsymbol{\mu}, \kappa) = C_{p}(\kappa) \exp \left( {\kappa \boldsymbol{\mu}^\mathsf{T} \mathbf{x} } \right),$$
where $\kappa \ge 0$,...
- 287