2
votes
0answers
65 views

Lower convex envelope of a function involving entropy

Suppose two discrete random variables $X$ and $Y$ defined on finite sets $\mathcal{X}$ and $\mathcal{Y}$ are given and also suppose the conditional distribution $P_{Y|X}$ (i.e, channel) is fixed. We ...
16
votes
0answers
493 views

conjectures regarding a new Renyi information quantity

In a recent paper http://arxiv.org/abs/1403.6102, we defined a quantity that we called the "Renyi conditional mutual information" and investigated several of its properties. We have some open ...
1
vote
1answer
125 views

Mutual information decrease with coarse-graining

Let $X,A,Y,B,C,D$ be random binary variables. $D$ is independent from $X,A,C$ and $C$ is independent from $Y,B,D$. Is it true that: If $I(Y:B|D=0)\leq \epsilon$ then $I(X\oplus Y:A\oplus ...
1
vote
0answers
80 views

Two questions on conditional independence [closed]

I have two simple questions about conditional independence. Suppose two random variables $X$ and $Y$ are independent. Then I want to show that they are independent given any arbitrary random ...
0
votes
0answers
67 views

An attempt to solve “Maximization of a total variation distance subject to another total variation distance in Markov chain”

I have been trying to solve Maximization of a total variation distance subject to another total variation distance in Markov chain. As a recall, suppose we have a pair of correlated random variables ...
3
votes
0answers
163 views

Maximization of a total variation distance subject to another total variation distance in Markov chain

Suppose two dependent random variables $X$ and $V$ from finite alphabets $\mathcal{V}$ and $\mathcal{X}$ with known joint and marginal distributions are given. Let $P_{XV}$ and $P_X$ and $P_V$ are the ...
5
votes
0answers
170 views

Maximizing Renyi entropy for a certain channel

The channel under consideration is $T = A + B$, where $A$ and $B$ take on values in $\{0, 1\}$ according to a probability mass function. Let (joint) random vector $(A_1, A_2,\ldots, A_n)$ be denoted ...
2
votes
0answers
130 views

An optimization in Markov Chain

We are given two correlated random variables $V$ and $X$ supported over a finite alphabets $\mathcal{V}$ and $\mathcal{X}$. Suppose the marginal $P_V$ and conditional distribution $P_{X|V}$ are ...
3
votes
1answer
365 views

Simple reason that a mathematician cannot do better than random when guessing contents of a box?

I have a question about the finite analog of the puzzle proposed here involving mathematicians guessing the contents of boxes. Specifically, suppose there are $k$ unopened boxes each containing a ...
2
votes
2answers
287 views

What is the maximum entropy distribution on the natural numbers?

On the reals $\mathbb{R}$, the maximum entropy distribution with a given mean and variance is the Gaussian distribution. Let $\mu, \sigma > 0$. What is the maximum entropy distribution on the ...
15
votes
1answer
662 views

Gini Coefficient and Renyi Entropy

Gini coefficient (aka Gini Index) is a quantity used in economics to describe income inequality. It is 0 for uniformly distributed income, and approaches 1 when all income is in hands of one ...
1
vote
0answers
107 views

expected inverse of circulant plus random diagonal

I have a deterministic circulant matrix $R$ and a random diagonal matrix $X$ where all elements are IID and positive. I need to determine the expected inverse of $R+X$, that is: Evaluate, in closed, ...
1
vote
1answer
196 views

Size of KL-divergence neighbourhoods

I am new here. I was reading another post here and this got me wondering what can be said about the size of the following kl divergence neighborhoods. Consider these two kl-divergence neighbourhood ...
1
vote
1answer
159 views

Set of distributions that minimize KL divergence,

Assuming that $p,q$ are probability distributions defined on the same support $\{x_i\}_{0 \leq i \leq n}$, $\epsilon$ a small real number, and $D_{KL}$ the Kullback-Leibler divergence, is there a ...
2
votes
0answers
93 views

Information theoretic privacy and distance of probability measures!

I came across the notion of information theoretic privacy in the paper of Yamamoto ("A source coding problem for sources with additional outputs to keep secret from the receiver or wiretappers "). The ...
2
votes
0answers
84 views

order of convergence of the conditional entropy (3)

I'm sorry for having open two questions which have been solved by elementary counter-examples provided by @AnthonyQuas. Actually I'm not an expert in information theory and I expected that a positive ...
1
vote
1answer
93 views

order of convergence of the conditional entropy (2)

Let $X_n$ be a random variable distributed on $A_n:=\{1, \ldots, n\}$ and $g_n\colon A_n \to A_n$ such that $\Pr\big(X_n \neq g_n(X_n)\big) \to 0$. Putting $Y_n=g_n(X_n)$, then by Fano's inequality ...
5
votes
1answer
81 views

a measure of difference for arrangements of sphere points

Suppose one has a distribution of $N$ points on the sphere. Is there an agreed upon metric for the difference of this distribution and $N$ equidistant points on the sphere? To me entropy seems like ...
16
votes
1answer
838 views

An Entropy Inequality (generalized)

Let $X,Y$ be probability measures on $\{1,2,\dots,n\}$. For $0\le \alpha \le 1$, set $K=\sum_i X(i)^\alpha Y(i)^{1-\alpha}$ so that $Z:=\frac{1}{K}X^\alpha Y^{1-\alpha}$ is also a probability ...
29
votes
1answer
2k views

An Entropy Inequality

Let $X,Y$ be probability measures on $\{1,2,\dots,n\}$, and set $K=\sum_i\sqrt{X(i)Y(i)}$ so that $Z:=\frac{1}{K}\sqrt{XY}$ is also a probability measure on $\{1,2,\dots,n\}$. How can we prove the ...
4
votes
3answers
435 views

Incremental entropy computation

After a quick internet search I found no method for incremental entropy computation. Question 1 Let $\{x_i\}_{i=1}^n$ and $\{x_i\}_{i=1+n}^{n+m}$ be two samples and let $S_i^j:=\sum_{k=i}^j x_k$. ...
3
votes
0answers
161 views

Axioms for Mutual Information

I am interesting in axiomatic justifications for concepts in information theory. I have found many axiomatizations for Shannon's entropy and for the Kullback-Leibler divergence, as well as their ...
2
votes
1answer
262 views

A Johnson-Lindenstrauss lemma for finite fields?

Given $m$ points in $\mathbb{R}^N$, the Johnson-Lindenstrauss lemma guarantees the existence of a linear operator $\mathbb{R}^N\rightarrow\mathbb{R}^n$ that nearly preserves pairwise distances between ...
5
votes
0answers
155 views

Chernoff bound in the not-quite-sub-exponential case

In Terry Tao's notes on Concentration of measure, Exercise 7 indicates that the Chernoff bound can be generalized to sub-exponential random variables: ...
2
votes
1answer
172 views

KL divergence(s) comparison,

Hi, $P_1$, $P_2$, $P_3$ are probability distributions defined on the same support. Knowing that $H(P_1) < H(P_2) < H(P_3)$, can we compare $D_{KL}(P_2,P_1)$ and $D_{KL}(P_3,P_1)$ ? (H is the ...
1
vote
1answer
252 views

Calculate channel capacity of general channel under constraint

Hi! Given a conditional distribution $P_{Y|X}$ I'd like to find the prior distribution $P_X$ that maximizes the mutual information $I(X;Y)$ with $P_Y(y)=\int P_{Y|X}(y|x)P_X(x)\text{dx}$ (this ...
2
votes
1answer
264 views

Derivative of a random process

Consider $w(t)$ as Guassian random process, with $w(t)$ being $\mathcal{N}(\mu,\sigma)$ and i.i.d for all t. I consider applying a (stochastic)derivative operation to the random process. What is the ...
8
votes
0answers
225 views

Uncertainty principle in Entropy terms

Math Questions: Consider Hilbert space $L_2(\mathbb{R})$ with a standard norm $ ||\psi|| = ( \int_{\mathbb{R}}{ |\psi(t)|^2 dt } )^{1/2}, $ and Fourier transform $ (F\psi)(\xi) = ...
2
votes
1answer
144 views

Moments of random matrices - when are they finite

I need to evaluate the moment $$\mathbb{E} (AX)^n,$$ where A is an NxN Hermitian square matrix, and X is $$X=ZZ^{\ast},$$ where $Z=\mu+Y$, where $\mu$ is mean of $Z$ and $Y$ is a zero-mean complex ...
2
votes
0answers
70 views

“Soft” Voronoi cells or statistical criterias

It is probably some basic statistics question, but... Informally 1: How to choose "criteria", such that it will guarantee that error decision probability is less than "epsilon", and maximize ...
3
votes
0answers
122 views

What distribution(s) of delays make(s) timing attacks hardest?

$H$ is (Shannon) entropy. In terms of the positive real number $t$, what distribution(s) $\hspace{.01 in}X$ on $\:[0\hspace{.005 in},\hspace{-0.03 in}\scriptsize+\normalsize\infty\hspace{-0.02 ...
5
votes
2answers
474 views

Sufficient Condition for Exponential Decay in Chernoff Bound (Large Deviations)

Let $X_i$ ($i=1,...,n$) be a sequence of independent and identically distributed random variables. Denote $\mu=\mathbb{E}[X_i]$ and $S_n=\frac{1}{n}\sum_{i=1}^nX_i$. This question concerns the tail ...
3
votes
1answer
236 views

convex combination of two covariance estimates

I am interested in covaraince matrix estimation. In brief: I have two estimates of the covariance matrix, and now I want to form a bona fide convex combination of the two. Background: I have studied ...
3
votes
0answers
128 views

Find a minimum entropy code for a simple gibbs random field.

Just to make precise what I am talking about, I will include the definition of a minimum entropy code. I will then define the precise markov random field I am asking about. In the rest of this ...
4
votes
2answers
347 views

Proving that a complicated function is eventually concave

I have a function $f:\mathbb{R}^+ \to \mathbb{R}^+$ that I want to prove is eventually concave - i.e. that there exists $\gamma _0 > 0$ such that for every $\gamma>\gamma_0$, $f(\gamma)$ is ...
4
votes
2answers
256 views

Maximum entropy priors in infinite dimensional spaces

Is there an extension of maximum entropy probability distributions for function spaces? For $\mathbb{R}^n$ and discrete spaces, there is much literature about this problem under names such as ...
0
votes
1answer
182 views

MMSE estimator expressed through cumulants

I have a linear model $$Y=HX+N,$$ where $H$ is a matrix and $X$ are drawn from $p_X(X)$, and $N$ is Gaussian noise variates. Now, if $X$ is multivariate Gaussian, then a linear estimator ...
3
votes
1answer
1k views

With Huffman code, why do we still need Shannon code?

I'm studying information theory by myself. I'm confused about that since we already have Huffman code, which is the optimal code method, why are Shannon code and some other code still useful? I ...
1
vote
1answer
138 views

The degrees in a random subgraph

Fix some positive integers $N$ and $d_k$, $k=1,2,\dots$ with $N=\sum_{k=1}^\infty d_k$. Suppose you have a graph $G$ taken randomly uniformly among the set of all (unoriented) graphs with $N$ ...
0
votes
2answers
196 views

multivariate distributions unaffected by unitary transformations

Hi, In my research I reached some very nice results for IID complex Gaussian vectors $\bf{x}$. Now I realize that my results hold for any random vectors that are unaffected by a unitary map, i.e., ...
6
votes
2answers
176 views

Number of neigbour Voronoi cells for a random set of points on S^k or cube [-1, 1]^k?

Consider $S^k \subset R^{k+1} $. Sample $N$ points by say uniform distribution. (Example k=120, N=2^24, i.e. N>>k ). Consider Voronoi cell around each point. How many neighbours would a cell have ...
1
vote
0answers
53 views

Information amount of fuzzy data transfer [closed]

Suppose we have binary channel from which we are able to receive zeroes and ones. We also know apriory probability $p$ of receiving "1". Then we can calculate information amount of each digit $q$ we ...
2
votes
1answer
204 views

Error bounds for truncating a probability distribution based on the entropy?

Heuristic Background Consider a set of states labeled $n=1,2,...$ in order of non-increasing probability $p(n)$. The standard Shannon argument gives meaning to the entropy $S$ of $p$ in terms of the ...
1
vote
2answers
281 views

Measuring the independence between the components of a stochastic process

In a context of blind source separation (e.g. you want to extract the voice of a singer from a song), many approaches consist in maximizing the independence between the components of a certain ...
1
vote
1answer
373 views

Bounding Entropy in terms of KL-Divergence

Let $h(X)$ be the differential entropy of a continuous random variable $X$ with density $f$, and let $Y$ be another continuous random variable with density $g$. If $KL(X\mid\mid Y)$ is the ...
3
votes
0answers
92 views

Is a parametric family which is universally consistent for multiple quantiles impossible?

Suppose I am dead-set on using Bayesian inference on independent and identically distributed data, but I'm lazy and insist on using a parametric likelihood function come what may. I'd be reassured to ...
2
votes
1answer
419 views

Divergence between two random variables

I have two Gaussian random variables $X$ and $Y$, each of which is an estimator of an underlying quantity. I need to measure whether $Y$ is estimating something different than $X$. So if the mean of ...
2
votes
0answers
124 views

finding rank-3 tensors compatible with a rank-2 tensor projection

I am interested in the following problem: Consider a rank-3 symmetric tensor $\boldsymbol{\sigma}$ with $\sigma_{ijk}$ where $\sigma_{ijk}$ can be 0 or 1, and the symmetry is with respect to any ...
8
votes
2answers
552 views

Is this a situation where triple mutual information is always non-negative?

Suppose I have three identically-distributed homogeneous continuous-time discrete state space Markov chains $X_1(t), X_2(t), X_3(t)$, $t\geq 0$. They evolve independently but share a common random ...
1
vote
0answers
194 views

Use of a priori information

I'm reading a paper [R1] where the authors propose a MAP estimator of the phase noise and frequency offset. However equation (17), which I reproduce below, represents a challenging step and I would ...