# Tagged Questions

**1**

vote

**1**answer

126 views

### Mutual information decrease with coarse-graining

Let $X,A,Y,B,C,D$ be random binary variables. $D$ is independent from $X,A,C$ and $C$ is independent from $Y,B,D$.
Is it true that:
If $I(Y:B|D=0)\leq \epsilon$ then $I(X\oplus Y:A\oplus ...

**1**

vote

**0**answers

82 views

### Two questions on conditional independence [closed]

I have two simple questions about conditional independence.
Suppose two random variables $X$ and $Y$ are independent. Then I want to show that they are independent given any arbitrary random ...

**0**

votes

**0**answers

20 views

### How to generalize uncertainty coefficient to set-valued classes?

This question is the reason I asked How to estimate the entropy of a distribution on a power set?
Proficiency (AKA uncertainty coefficient) is an information-theoretic measure of predictor quality, ...

**1**

vote

**3**answers

111 views

### How to estimate the entropy of a distribution on a power set?

Given a probability distribution $(X,p)$, its entropy is defined as $H=-\sum_{x\in X} p(x)\log p(x)$.
Given a sample of observations $x_n,n=1..N$, one can estimate $p(x)=\frac{\#\{i:x_i=x\}}{N}$ and ...

**3**

votes

**0**answers

163 views

### Maximization of a total variation distance subject to another total variation distance in Markov chain

Suppose two dependent random variables $X$ and $V$ from finite alphabets $\mathcal{V}$ and $\mathcal{X}$ with known joint and marginal distributions are given. Let $P_{XV}$ and $P_X$ and $P_V$ are the ...

**8**

votes

**2**answers

334 views

### Inequality in information theory

I am reading the paper "chain independence and common information" (http://ttic.uchicago.edu/~yury/papers/independ.pdf). In this paper, an inequality is used several times (without proof) which looks ...

**2**

votes

**2**answers

289 views

### What is the maximum entropy distribution on the natural numbers?

On the reals $\mathbb{R}$, the maximum entropy distribution with a given mean and variance is the Gaussian distribution.
Let $\mu, \sigma > 0$. What is the maximum entropy distribution on the ...

**15**

votes

**1**answer

669 views

### Gini Coefficient and Renyi Entropy

Gini coefficient (aka Gini Index) is a quantity used in economics to describe income inequality. It is 0 for uniformly distributed income, and approaches 1 when all income is in hands of one ...

**1**

vote

**1**answer

196 views

### Size of KL-divergence neighbourhoods

I am new here. I was reading another
post
here and this got me wondering what can be said about the size of the following kl divergence neighborhoods.
Consider these two kl-divergence neighbourhood ...

**4**

votes

**3**answers

437 views

### Incremental entropy computation

After a quick internet search I found no method for incremental entropy computation.
Question 1
Let $\{x_i\}_{i=1}^n$ and $\{x_i\}_{i=1+n}^{n+m}$ be two samples and let $S_i^j:=\sum_{k=i}^j x_k$. ...

**2**

votes

**1**answer

267 views

### Derivative of a random process

Consider $w(t)$ as Guassian random process, with $w(t)$ being $\mathcal{N}(\mu,\sigma)$ and i.i.d for all t.
I consider applying a (stochastic)derivative operation to the random process. What is the ...

**2**

votes

**1**answer

144 views

### Moments of random matrices - when are they finite

I need to evaluate the moment
$$\mathbb{E} (AX)^n,$$ where A is an NxN Hermitian square matrix, and X is
$$X=ZZ^{\ast},$$ where
$Z=\mu+Y$, where $\mu$ is mean of $Z$ and $Y$ is a zero-mean complex ...

**2**

votes

**0**answers

70 views

### “Soft” Voronoi cells or statistical criterias

It is probably some basic statistics question, but...
Informally 1: How to choose "criteria", such that it will guarantee that error decision probability is less than "epsilon", and maximize ...

**3**

votes

**1**answer

238 views

### convex combination of two covariance estimates

I am interested in covaraince matrix estimation. In brief: I have two estimates of the covariance matrix, and now I want to form a bona fide convex combination of the two.
Background: I have studied ...

**3**

votes

**0**answers

129 views

### Find a minimum entropy code for a simple gibbs random field.

Just to make precise what I am talking about, I will include the definition of a minimum entropy code. I will then define the precise markov random field I am asking about.
In the rest of this ...

**1**

vote

**2**answers

281 views

### Measuring the independence between the components of a stochastic process

In a context of blind source separation (e.g. you want to extract the voice of a singer from a song), many approaches consist in maximizing the independence between the components of a certain ...

**2**

votes

**3**answers

319 views

### mutual information and minimal communication required for generating correlation

Let $X$,$Y$ be two stochastic variables with probability distribution $\rho(X,Y)$. The
mutual information, $I(X;Y)$, represents the information shared by the two variables. This intuitive ...

**3**

votes

**0**answers

93 views

### Is a parametric family which is universally consistent for multiple quantiles impossible?

Suppose I am dead-set on using Bayesian inference on independent and identically distributed data, but I'm lazy and insist on using a parametric likelihood function come what may. I'd be reassured to ...

**2**

votes

**0**answers

124 views

### finding rank-3 tensors compatible with a rank-2 tensor projection

I am interested in the following problem: Consider a rank-3 symmetric tensor $\boldsymbol{\sigma}$ with $\sigma_{ijk}$ where $\sigma_{ijk}$ can be 0 or 1, and the symmetry is with respect to any ...

**2**

votes

**0**answers

203 views

### Error exponent in hypothesis testing

In hypothesis testing, one must decide between two probability distributions $P_1(x)$ and $P_2(x)$ on a finite set $X$, after observing $n$ i.i.d. samples $x_1,...,x_n$ drawn from the unknown ...

**3**

votes

**0**answers

450 views

### On error probability bounds in Bayesian hypothesis testing

In the Bayesian version of (binary) hypothesis testing one has to decide which of two hypotheses $A$ and $B$ holds true. The two hypotheses are given prior probability $p(A)$ and $p(B)$, summing up to ...

**2**

votes

**0**answers

366 views

### How to calculate/approximate expectation of function of a binomial random variable?

Hi,
I am stuck at following problem in my research.
Suppose that $M=m$ is a random variable with binomial distribution with parameters $n,p$. The constants $r$ and $\gamma$ are greater than zero. ...

**0**

votes

**1**answer

911 views

### basic measure theory question - measure on the natural numbers [closed]

I am looking for a succinct way to describe a subset of the natural numbers which has ``measure zero" in the following sense: Let X_1 \subset X_2 \subset .... be any strictly nesting sequence of ...

**1**

vote

**1**answer

411 views

### Is there some generalization of the “Maximum Coverage Problem” for information in random variables ?

Background :
I'm trying to learn aspects of information theory that might be relevant for neural decoding. I am a student and do not have a strong formal background in mathematics, statistics, or ...

**0**

votes

**1**answer

408 views

### Information criteria for ridge regression

Hi -- is there any analogue or adjustment of, say, Schwartz Bayesian (or other) information criterion that would be applicable to model selection in ridge regression with a given ridge parameter ...

**5**

votes

**2**answers

599 views

### Convergence of an empirical distribution w.r.t. the Hellinger distance

Let $P$ be a probability distribution on a finite set $\mathcal{X}$ and let $X_1, X_2, \ldots, X_n$ be drawn i.i.d. according to $P$. Define the empirical distribution:
$\hat{P_n}(x) = \frac{1}{n} ...

**1**

vote

**1**answer

447 views

### Showing non-attainment of supremum

This is just an extension of my previous question Tightness of probabilty distributions
Let $\mathcal{P}(\mathbb{N})$ be the set of all PMF's on $\mathbb{N}=\{1,2,\dots \}$. Let $E$ be a convex ...

**2**

votes

**2**answers

740 views

### Tightness of probabilty distributions

Let $\mathcal{P}(\mathbb{N})$ be the set of all probability mass functions on $\mathbb{N}=\{1,2,\dots \}$. Let $E$ be a closed(with respect to pointwise convergence, or equivalently the total ...

**4**

votes

**0**answers

1k views

### Using Fisher Information to bound KL divergence

Is it possible to use Fisher Information at p to get a useful upper bound on KL(q,p)?
KL(q,p) is known as Kullback-Liebler divergence and is defined for discrete distributions over k outcomes as ...

**10**

votes

**2**answers

2k views

### Bounding sum of multinomial coefficients by highest entropy one

When does the following hold?
$\sum_{(i_1,\ldots,i_k)\in E}
\frac{n!}{i_1! \ldots i_k!}
\le \exp(n H^*)$
Where
$H^*=\max_{(i_1,\ldots,i_k)\in E} -(\frac{i_1}{n}\log \frac{i_1}{n}+\ldots ...

**5**

votes

**3**answers

2k views

### Sanov's Theorem and Chernoff bound

Sanov's Theorem (p.292, Thomas/Cover "Elements of Information Theory" (1991)) says that probability of a hypothesis $E$ according to distribution $Q$ is bounded above by
$$(n+1)^k \exp (-n D(P^* ...

**3**

votes

**0**answers

695 views

### Compressed Sensing with an Unusual Basis

I'm wondering if compressed sensing can be applied to a problem I have in the way I describe, and also whether it should be applied to this problem (or whether it's simply the wrong tool).
I have a ...

**5**

votes

**2**answers

986 views

### Inequality involving probability measures [closed]

I have been working on a problem(alternate minimization) where I want to establish an inequality in which I am stuck.
An $\alpha$- parameterized version of the divergence(Kullback-Leibler) takes the ...

**4**

votes

**1**answer

491 views

### O(n^2) algorithm to approximate the sum of the log of the singular values of a matrix

Given an $M \times N$ matrix of rank $N$ ($M \ge N$) with $i^{th}$ singular value $\sigma_i$, does their exist an $O(M^2)$ algorithm for approximating the sum $ H =\sum_{i=1}^N \log(\sigma_i)$ with ...