0
votes
0answers
63 views

An attempt to solve “Maximization of a total variation distance subject to another total variation distance in Markov chain”

I have been trying to solve Maximization of a total variation distance subject to another total variation distance in Markov chain. As a recall, suppose we have a pair of correlated random variables ...
3
votes
0answers
148 views

Maximization of a total variation distance subject to another total variation distance in Markov chain

Suppose two dependent random variables $X$ and $V$ from finite alphabets $\mathcal{V}$ and $\mathcal{X}$ with known joint and marginal distributions are given. Let $P_{XV}$ and $P_X$ and $P_V$ are the ...
4
votes
0answers
151 views

Maximizing Renyi entropy for a certain channel

The channel under consideration is $T = A + B$, where $A$ and $B$ take on values in $\{0, 1\}$ according to a probability mass function. Let (joint) random vector $(A_1, A_2,\ldots, A_n)$ be denoted ...
1
vote
0answers
122 views

An optimization in Markov Chain

We are given two correlated random variables $V$ and $X$ supported over a finite alphabets $\mathcal{V}$ and $\mathcal{X}$. Suppose the marginal $P_V$ and conditional distribution $P_{X|V}$ are ...
3
votes
1answer
351 views

Simple reason that a mathematician cannot do better than random when guessing contents of a box?

I have a question about the finite analog of the puzzle proposed here involving mathematicians guessing the contents of boxes. Specifically, suppose there are $k$ unopened boxes each containing a ...
2
votes
2answers
273 views

What is the maximum entropy distribution on the natural numbers?

On the reals $\mathbb{R}$, the maximum entropy distribution with a given mean and variance is the Gaussian distribution. Let $\mu, \sigma > 0$. What is the maximum entropy distribution on the ...
15
votes
1answer
632 views

Gini Coefficient and Renyi Entropy

Gini coefficient (aka Gini Index) is a quantity used in economics to describe income inequality. It is 0 for uniformly distributed income, and approaches 1 when all income is in hands of one ...
1
vote
0answers
102 views

expected inverse of circulant plus random diagonal

I have a deterministic circulant matrix $R$ and a random diagonal matrix $X$ where all elements are IID and positive. I need to determine the expected inverse of $R+X$, that is: Evaluate, in closed, ...
0
votes
1answer
160 views

Size of KL-divergence neighbourhoods

I am new here. I was reading another post here and this got me wondering what can be said about the size of the following kl divergence neighborhoods. Consider these two kl-divergence neighbourhood ...
1
vote
1answer
152 views

Set of distributions that minimize KL divergence,

Assuming that $p,q$ are probability distributions defined on the same support $\{x_i\}_{0 \leq i \leq n}$, $\epsilon$ a small real number, and $D_{KL}$ the Kullback-Leibler divergence, is there a ...
2
votes
0answers
88 views

Information theoretic privacy and distance of probability measures!

I came across the notion of information theoretic privacy in the paper of Yamamoto ("A source coding problem for sources with additional outputs to keep secret from the receiver or wiretappers "). The ...
2
votes
0answers
79 views

order of convergence of the conditional entropy (3)

I'm sorry for having open two questions which have been solved by elementary counter-examples provided by @AnthonyQuas. Actually I'm not an expert in information theory and I expected that a positive ...
1
vote
1answer
89 views

order of convergence of the conditional entropy (2)

Let $X_n$ be a random variable distributed on $A_n:=\{1, \ldots, n\}$ and $g_n\colon A_n \to A_n$ such that $\Pr\big(X_n \neq g_n(X_n)\big) \to 0$. Putting $Y_n=g_n(X_n)$, then by Fano's inequality ...
5
votes
1answer
77 views

a measure of difference for arrangements of sphere points

Suppose one has a distribution of $N$ points on the sphere. Is there an agreed upon metric for the difference of this distribution and $N$ equidistant points on the sphere? To me entropy seems like ...
13
votes
1answer
677 views

An Entropy Inequality (generalized)

Let $X,Y$ be probability measures on $\{1,2,\dots,n\}$. For $0\le \alpha \le 1$, set $K=\sum_i X(i)^\alpha Y(i)^{1-\alpha}$ so that $Z:=\frac{1}{K}X^\alpha Y^{1-\alpha}$ is also a probability ...
28
votes
1answer
2k views

An Entropy Inequality

Let $X,Y$ be probability measures on $\{1,2,\dots,n\}$, and set $K=\sum_i\sqrt{X(i)Y(i)}$ so that $Z:=\frac{1}{K}\sqrt{XY}$ is also a probability measure on $\{1,2,\dots,n\}$. How can we prove the ...
4
votes
3answers
409 views

Incremental entropy computation

After a quick internet search I found no method for incremental entropy computation. Question 1 Let $\{x_i\}_{i=1}^n$ and $\{x_i\}_{i=1+n}^{n+m}$ be two samples and let $S_i^j:=\sum_{k=i}^j x_k$. ...
3
votes
0answers
151 views

Axioms for Mutual Information

I am interesting in axiomatic justifications for concepts in information theory. I have found many axiomatizations for Shannon's entropy and for the Kullback-Leibler divergence, as well as their ...
2
votes
1answer
257 views

A Johnson-Lindenstrauss lemma for finite fields?

Given $m$ points in $\mathbb{R}^N$, the Johnson-Lindenstrauss lemma guarantees the existence of a linear operator $\mathbb{R}^N\rightarrow\mathbb{R}^n$ that nearly preserves pairwise distances between ...
5
votes
0answers
152 views

Chernoff bound in the not-quite-sub-exponential case

In Terry Tao's notes on Concentration of measure, Exercise 7 indicates that the Chernoff bound can be generalized to sub-exponential random variables: ...
2
votes
1answer
162 views

KL divergence(s) comparison,

Hi, $P_1$, $P_2$, $P_3$ are probability distributions defined on the same support. Knowing that $H(P_1) < H(P_2) < H(P_3)$, can we compare $D_{KL}(P_2,P_1)$ and $D_{KL}(P_3,P_1)$ ? (H is the ...
1
vote
1answer
210 views

Calculate channel capacity of general channel under constraint

Hi! Given a conditional distribution $P_{Y|X}$ I'd like to find the prior distribution $P_X$ that maximizes the mutual information $I(X;Y)$ with $P_Y(y)=\int P_{Y|X}(y|x)P_X(x)\text{dx}$ (this ...
1
vote
1answer
238 views

Derivative of a random process

Consider $w(t)$ as Guassian random process, with $w(t)$ being $\mathcal{N}(\mu,\sigma)$ and i.i.d for all t. I consider applying a (stochastic)derivative operation to the random process. What is the ...
7
votes
0answers
201 views

Uncertainty principle in Entropy terms

Math Questions: Consider Hilbert space $L_2(\mathbb{R})$ with a standard norm $ ||\psi|| = ( \int_{\mathbb{R}}{ |\psi(t)|^2 dt } )^{1/2}, $ and Fourier transform $ (F\psi)(\xi) = ...
2
votes
1answer
143 views

Moments of random matrices - when are they finite

I need to evaluate the moment $$\mathbb{E} (AX)^n,$$ where A is an NxN Hermitian square matrix, and X is $$X=ZZ^{\ast},$$ where $Z=\mu+Y$, where $\mu$ is mean of $Z$ and $Y$ is a zero-mean complex ...
2
votes
0answers
68 views

“Soft” Voronoi cells or statistical criterias

It is probably some basic statistics question, but... Informally 1: How to choose "criteria", such that it will guarantee that error decision probability is less than "epsilon", and maximize ...
3
votes
0answers
120 views

What distribution(s) of delays make(s) timing attacks hardest?

$H$ is (Shannon) entropy. In terms of the positive real number $t$, what distribution(s) $\hspace{.01 in}X$ on $\:[0\hspace{.005 in},\hspace{-0.03 in}\scriptsize+\normalsize\infty\hspace{-0.02 ...
5
votes
2answers
455 views

Sufficient Condition for Exponential Decay in Chernoff Bound (Large Deviations)

Let $X_i$ ($i=1,...,n$) be a sequence of independent and identically distributed random variables. Denote $\mu=\mathbb{E}[X_i]$ and $S_n=\frac{1}{n}\sum_{i=1}^nX_i$. This question concerns the tail ...
3
votes
1answer
223 views

convex combination of two covariance estimates

I am interested in covaraince matrix estimation. In brief: I have two estimates of the covariance matrix, and now I want to form a bona fide convex combination of the two. Background: I have studied ...
3
votes
0answers
126 views

Find a minimum entropy code for a simple gibbs random field.

Just to make precise what I am talking about, I will include the definition of a minimum entropy code. I will then define the precise markov random field I am asking about. In the rest of this ...
4
votes
2answers
342 views

Proving that a complicated function is eventually concave

I have a function $f:\mathbb{R}^+ \to \mathbb{R}^+$ that I want to prove is eventually concave - i.e. that there exists $\gamma _0 > 0$ such that for every $\gamma>\gamma_0$, $f(\gamma)$ is ...
4
votes
2answers
252 views

Maximum entropy priors in infinite dimensional spaces

Is there an extension of maximum entropy probability distributions for function spaces? For $\mathbb{R}^n$ and discrete spaces, there is much literature about this problem under names such as ...
0
votes
1answer
178 views

MMSE estimator expressed through cumulants

I have a linear model $$Y=HX+N,$$ where $H$ is a matrix and $X$ are drawn from $p_X(X)$, and $N$ is Gaussian noise variates. Now, if $X$ is multivariate Gaussian, then a linear estimator ...
3
votes
1answer
1k views

With Huffman code, why do we still need Shannon code?

I'm studying information theory by myself. I'm confused about that since we already have Huffman code, which is the optimal code method, why are Shannon code and some other code still useful? I ...
1
vote
1answer
128 views

The degrees in a random subgraph

Fix some positive integers $N$ and $d_k$, $k=1,2,\dots$ with $N=\sum_{k=1}^\infty d_k$. Suppose you have a graph $G$ taken randomly uniformly among the set of all (unoriented) graphs with $N$ ...
0
votes
2answers
176 views

multivariate distributions unaffected by unitary transformations

Hi, In my research I reached some very nice results for IID complex Gaussian vectors $\bf{x}$. Now I realize that my results hold for any random vectors that are unaffected by a unitary map, i.e., ...
6
votes
2answers
169 views

Number of neigbour Voronoi cells for a random set of points on S^k or cube [-1, 1]^k?

Consider $S^k \subset R^{k+1} $. Sample $N$ points by say uniform distribution. (Example k=120, N=2^24, i.e. N>>k ). Consider Voronoi cell around each point. How many neighbours would a cell have ...
1
vote
0answers
52 views

Information amount of fuzzy data transfer [closed]

Suppose we have binary channel from which we are able to receive zeroes and ones. We also know apriory probability $p$ of receiving "1". Then we can calculate information amount of each digit $q$ we ...
2
votes
1answer
203 views

Error bounds for truncating a probability distribution based on the entropy?

Heuristic Background Consider a set of states labeled $n=1,2,...$ in order of non-increasing probability $p(n)$. The standard Shannon argument gives meaning to the entropy $S$ of $p$ in terms of the ...
1
vote
2answers
275 views

Measuring the independence between the components of a stochastic process

In a context of blind source separation (e.g. you want to extract the voice of a singer from a song), many approaches consist in maximizing the independence between the components of a certain ...
1
vote
1answer
354 views

Bounding Entropy in terms of KL-Divergence

Let $h(X)$ be the differential entropy of a continuous random variable $X$ with density $f$, and let $Y$ be another continuous random variable with density $g$. If $KL(X\mid\mid Y)$ is the ...
3
votes
0answers
91 views

Is a parametric family which is universally consistent for multiple quantiles impossible?

Suppose I am dead-set on using Bayesian inference on independent and identically distributed data, but I'm lazy and insist on using a parametric likelihood function come what may. I'd be reassured to ...
2
votes
1answer
396 views

Divergence between two random variables

I have two Gaussian random variables $X$ and $Y$, each of which is an estimator of an underlying quantity. I need to measure whether $Y$ is estimating something different than $X$. So if the mean of ...
2
votes
0answers
123 views

finding rank-3 tensors compatible with a rank-2 tensor projection

I am interested in the following problem: Consider a rank-3 symmetric tensor $\boldsymbol{\sigma}$ with $\sigma_{ijk}$ where $\sigma_{ijk}$ can be 0 or 1, and the symmetry is with respect to any ...
8
votes
2answers
541 views

Is this a situation where triple mutual information is always non-negative?

Suppose I have three identically-distributed homogeneous continuous-time discrete state space Markov chains $X_1(t), X_2(t), X_3(t)$, $t\geq 0$. They evolve independently but share a common random ...
1
vote
0answers
191 views

Use of a priori information

I'm reading a paper [R1] where the authors propose a MAP estimator of the phase noise and frequency offset. However equation (17), which I reproduce below, represents a challenging step and I would ...
2
votes
0answers
346 views

How to calculate/approximate expectation of function of a binomial random variable?

Hi, I am stuck at following problem in my research. Suppose that $M=m$ is a random variable with binomial distribution with parameters $n,p$. The constants $r$ and $\gamma$ are greater than zero. ...
0
votes
0answers
621 views

Calculate entropy for a set of data

Hi; I am not really a maths person and I have a question regarding shannon entropy. I have different datasets which only consists of three letter(I,N,M) such as : dataset1: {I,I,N,M,I,I} dataset2: ...
2
votes
1answer
324 views

metric for signal to noise ratio in communication systems

I'm not quite sure about how to define a good measure of the quality of a communication channel with fading and interference. Let us assume the simplest case, where a node in a network receives the ...
9
votes
3answers
884 views

Entropy of a measure

Let $\mu$ be a probability measure on a set of $n$ elements and let $p_i$ be the measure of the $i$-th element. Its Shannon entropy is defined by $$ E(\mu)=-\sum_{i=1}^np_i\log(p_i) $$ with the ...