1
vote
1answer
125 views

Mutual information decrease with coarse-graining

Let $X,A,Y,B,C,D$ be random binary variables. $D$ is independent from $X,A,C$ and $C$ is independent from $Y,B,D$. Is it true that: If $I(Y:B|D=0)\leq \epsilon$ then $I(X\oplus Y:A\oplus ...
1
vote
0answers
80 views

Two questions on conditional independence [closed]

I have two simple questions about conditional independence. Suppose two random variables $X$ and $Y$ are independent. Then I want to show that they are independent given any arbitrary random ...
0
votes
0answers
18 views

How to generalize uncertainty coefficient to set-valued classes?

This question is the reason I asked How to estimate the entropy of a distribution on a power set? Proficiency (AKA uncertainty coefficient) is an information-theoretic measure of predictor quality, ...
1
vote
3answers
107 views

How to estimate the entropy of a distribution on a power set?

Given a probability distribution $(X,p)$, its entropy is defined as $H=-\sum_{x\in X} p(x)\log p(x)$. Given a sample of observations $x_n,n=1..N$, one can estimate $p(x)=\frac{\#\{i:x_i=x\}}{N}$ and ...
3
votes
0answers
163 views

Maximization of a total variation distance subject to another total variation distance in Markov chain

Suppose two dependent random variables $X$ and $V$ from finite alphabets $\mathcal{V}$ and $\mathcal{X}$ with known joint and marginal distributions are given. Let $P_{XV}$ and $P_X$ and $P_V$ are the ...
8
votes
2answers
332 views

Inequality in information theory

I am reading the paper "chain independence and common information" (http://ttic.uchicago.edu/~yury/papers/independ.pdf). In this paper, an inequality is used several times (without proof) which looks ...
2
votes
2answers
287 views

What is the maximum entropy distribution on the natural numbers?

On the reals $\mathbb{R}$, the maximum entropy distribution with a given mean and variance is the Gaussian distribution. Let $\mu, \sigma > 0$. What is the maximum entropy distribution on the ...
15
votes
1answer
662 views

Gini Coefficient and Renyi Entropy

Gini coefficient (aka Gini Index) is a quantity used in economics to describe income inequality. It is 0 for uniformly distributed income, and approaches 1 when all income is in hands of one ...
1
vote
1answer
196 views

Size of KL-divergence neighbourhoods

I am new here. I was reading another post here and this got me wondering what can be said about the size of the following kl divergence neighborhoods. Consider these two kl-divergence neighbourhood ...
4
votes
3answers
435 views

Incremental entropy computation

After a quick internet search I found no method for incremental entropy computation. Question 1 Let $\{x_i\}_{i=1}^n$ and $\{x_i\}_{i=1+n}^{n+m}$ be two samples and let $S_i^j:=\sum_{k=i}^j x_k$. ...
2
votes
1answer
264 views

Derivative of a random process

Consider $w(t)$ as Guassian random process, with $w(t)$ being $\mathcal{N}(\mu,\sigma)$ and i.i.d for all t. I consider applying a (stochastic)derivative operation to the random process. What is the ...
2
votes
1answer
144 views

Moments of random matrices - when are they finite

I need to evaluate the moment $$\mathbb{E} (AX)^n,$$ where A is an NxN Hermitian square matrix, and X is $$X=ZZ^{\ast},$$ where $Z=\mu+Y$, where $\mu$ is mean of $Z$ and $Y$ is a zero-mean complex ...
2
votes
0answers
70 views

“Soft” Voronoi cells or statistical criterias

It is probably some basic statistics question, but... Informally 1: How to choose "criteria", such that it will guarantee that error decision probability is less than "epsilon", and maximize ...
3
votes
1answer
236 views

convex combination of two covariance estimates

I am interested in covaraince matrix estimation. In brief: I have two estimates of the covariance matrix, and now I want to form a bona fide convex combination of the two. Background: I have studied ...
3
votes
0answers
128 views

Find a minimum entropy code for a simple gibbs random field.

Just to make precise what I am talking about, I will include the definition of a minimum entropy code. I will then define the precise markov random field I am asking about. In the rest of this ...
1
vote
2answers
281 views

Measuring the independence between the components of a stochastic process

In a context of blind source separation (e.g. you want to extract the voice of a singer from a song), many approaches consist in maximizing the independence between the components of a certain ...
2
votes
3answers
318 views

mutual information and minimal communication required for generating correlation

Let $X$,$Y$ be two stochastic variables with probability distribution $\rho(X,Y)$. The mutual information, $I(X;Y)$, represents the information shared by the two variables. This intuitive ...
3
votes
0answers
92 views

Is a parametric family which is universally consistent for multiple quantiles impossible?

Suppose I am dead-set on using Bayesian inference on independent and identically distributed data, but I'm lazy and insist on using a parametric likelihood function come what may. I'd be reassured to ...
2
votes
0answers
124 views

finding rank-3 tensors compatible with a rank-2 tensor projection

I am interested in the following problem: Consider a rank-3 symmetric tensor $\boldsymbol{\sigma}$ with $\sigma_{ijk}$ where $\sigma_{ijk}$ can be 0 or 1, and the symmetry is with respect to any ...
2
votes
0answers
202 views

Error exponent in hypothesis testing

In hypothesis testing, one must decide between two probability distributions $P_1(x)$ and $P_2(x)$ on a finite set $X$, after observing $n$ i.i.d. samples $x_1,...,x_n$ drawn from the unknown ...
3
votes
0answers
449 views

On error probability bounds in Bayesian hypothesis testing

In the Bayesian version of (binary) hypothesis testing one has to decide which of two hypotheses $A$ and $B$ holds true. The two hypotheses are given prior probability $p(A)$ and $p(B)$, summing up to ...
2
votes
0answers
366 views

How to calculate/approximate expectation of function of a binomial random variable?

Hi, I am stuck at following problem in my research. Suppose that $M=m$ is a random variable with binomial distribution with parameters $n,p$. The constants $r$ and $\gamma$ are greater than zero. ...
0
votes
1answer
910 views

basic measure theory question - measure on the natural numbers [closed]

I am looking for a succinct way to describe a subset of the natural numbers which has ``measure zero" in the following sense: Let X_1 \subset X_2 \subset .... be any strictly nesting sequence of ...
1
vote
1answer
408 views

Is there some generalization of the “Maximum Coverage Problem” for information in random variables ?

Background : I'm trying to learn aspects of information theory that might be relevant for neural decoding. I am a student and do not have a strong formal background in mathematics, statistics, or ...
0
votes
1answer
407 views

Information criteria for ridge regression

Hi -- is there any analogue or adjustment of, say, Schwartz Bayesian (or other) information criterion that would be applicable to model selection in ridge regression with a given ridge parameter ...
5
votes
2answers
594 views

Convergence of an empirical distribution w.r.t. the Hellinger distance

Let $P$ be a probability distribution on a finite set $\mathcal{X}$ and let $X_1, X_2, \ldots, X_n$ be drawn i.i.d. according to $P$. Define the empirical distribution: $\hat{P_n}(x) = \frac{1}{n} ...
1
vote
1answer
447 views

Showing non-attainment of supremum

This is just an extension of my previous question Tightness of probabilty distributions Let $\mathcal{P}(\mathbb{N})$ be the set of all PMF's on $\mathbb{N}=\{1,2,\dots \}$. Let $E$ be a convex ...
2
votes
2answers
739 views

Tightness of probabilty distributions

Let $\mathcal{P}(\mathbb{N})$ be the set of all probability mass functions on $\mathbb{N}=\{1,2,\dots \}$. Let $E$ be a closed(with respect to pointwise convergence, or equivalently the total ...
4
votes
0answers
1k views

Using Fisher Information to bound KL divergence

Is it possible to use Fisher Information at p to get a useful upper bound on KL(q,p)? KL(q,p) is known as Kullback-Liebler divergence and is defined for discrete distributions over k outcomes as ...
10
votes
2answers
2k views

Bounding sum of multinomial coefficients by highest entropy one

When does the following hold? $\sum_{(i_1,\ldots,i_k)\in E} \frac{n!}{i_1! \ldots i_k!} \le \exp(n H^*)$ Where $H^*=\max_{(i_1,\ldots,i_k)\in E} -(\frac{i_1}{n}\log \frac{i_1}{n}+\ldots ...
5
votes
3answers
2k views

Sanov's Theorem and Chernoff bound

Sanov's Theorem (p.292, Thomas/Cover "Elements of Information Theory" (1991)) says that probability of a hypothesis $E$ according to distribution $Q$ is bounded above by $$(n+1)^k \exp (-n D(P^* ...
3
votes
0answers
694 views

Compressed Sensing with an Unusual Basis

I'm wondering if compressed sensing can be applied to a problem I have in the way I describe, and also whether it should be applied to this problem (or whether it's simply the wrong tool). I have a ...
5
votes
2answers
982 views

Inequality involving probability measures [closed]

I have been working on a problem(alternate minimization) where I want to establish an inequality in which I am stuck. An $\alpha$- parameterized version of the divergence(Kullback-Leibler) takes the ...
4
votes
1answer
491 views

O(n^2) algorithm to approximate the sum of the log of the singular values of a matrix

Given an $M \times N$ matrix of rank $N$ ($M \ge N$) with $i^{th}$ singular value $\sigma_i$, does their exist an $O(M^2)$ algorithm for approximating the sum $ H =\sum_{i=1}^N \log(\sigma_i)$ with ...