# Tagged Questions

**0**

votes

**0**answers

63 views

### An attempt to solve “Maximization of a total variation distance subject to another total variation distance in Markov chain”

I have been trying to solve Maximization of a total variation distance subject to another total variation distance in Markov chain. As a recall, suppose we have a pair of correlated random variables ...

**3**

votes

**0**answers

148 views

### Maximization of a total variation distance subject to another total variation distance in Markov chain

Suppose two dependent random variables $X$ and $V$ from finite alphabets $\mathcal{V}$ and $\mathcal{X}$ with known joint and marginal distributions are given. Let $P_{XV}$ and $P_X$ and $P_V$ are the ...

**4**

votes

**0**answers

151 views

### Maximizing Renyi entropy for a certain channel

The channel under consideration is $T = A + B$, where $A$ and $B$ take on values in $\{0, 1\}$ according to a probability mass function. Let (joint) random vector $(A_1, A_2,\ldots, A_n)$ be denoted ...

**1**

vote

**0**answers

122 views

### An optimization in Markov Chain

We are given two correlated random variables $V$ and $X$ supported over a finite alphabets $\mathcal{V}$ and $\mathcal{X}$. Suppose the marginal $P_V$ and conditional distribution $P_{X|V}$ are ...

**3**

votes

**1**answer

351 views

### Simple reason that a mathematician cannot do better than random when guessing contents of a box?

I have a question about the finite analog of the puzzle proposed here involving mathematicians guessing the contents of boxes.
Specifically, suppose there are $k$ unopened boxes each containing a ...

**2**

votes

**2**answers

273 views

### What is the maximum entropy distribution on the natural numbers?

On the reals $\mathbb{R}$, the maximum entropy distribution with a given mean and variance is the Gaussian distribution.
Let $\mu, \sigma > 0$. What is the maximum entropy distribution on the ...

**15**

votes

**1**answer

632 views

### Gini Coefficient and Renyi Entropy

Gini coefficient (aka Gini Index) is a quantity used in economics to describe income inequality. It is 0 for uniformly distributed income, and approaches 1 when all income is in hands of one ...

**1**

vote

**0**answers

102 views

### expected inverse of circulant plus random diagonal

I have a deterministic circulant matrix $R$ and a random diagonal matrix $X$ where all elements are IID and positive.
I need to determine the expected inverse of $R+X$, that is: Evaluate, in closed, ...

**0**

votes

**1**answer

160 views

### Size of KL-divergence neighbourhoods

I am new here. I was reading another
post
here and this got me wondering what can be said about the size of the following kl divergence neighborhoods.
Consider these two kl-divergence neighbourhood ...

**1**

vote

**1**answer

152 views

### Set of distributions that minimize KL divergence,

Assuming that $p,q$ are probability distributions defined on the same support $\{x_i\}_{0 \leq i \leq n}$, $\epsilon$ a small real number, and $D_{KL}$ the Kullback-Leibler divergence,
is there a ...

**2**

votes

**0**answers

88 views

### Information theoretic privacy and distance of probability measures!

I came across the notion of information theoretic privacy in the paper of Yamamoto ("A source coding problem for sources with additional outputs to keep secret from the receiver or wiretappers "). The ...

**2**

votes

**0**answers

79 views

### order of convergence of the conditional entropy (3)

I'm sorry for having open two questions which have been solved by elementary counter-examples provided by @AnthonyQuas. Actually I'm not an expert in information theory and I expected that a positive ...

**1**

vote

**1**answer

89 views

### order of convergence of the conditional entropy (2)

Let $X_n$ be a random variable distributed on $A_n:=\{1, \ldots, n\}$ and $g_n\colon A_n \to A_n$ such that $\Pr\big(X_n \neq g_n(X_n)\big) \to 0$. Putting $Y_n=g_n(X_n)$, then by Fano's inequality ...

**5**

votes

**1**answer

77 views

### a measure of difference for arrangements of sphere points

Suppose one has a distribution of $N$ points on the sphere. Is there an agreed upon metric for the difference of this distribution and $N$ equidistant points on the sphere? To me entropy seems like ...

**13**

votes

**1**answer

677 views

### An Entropy Inequality (generalized)

Let $X,Y$ be probability measures on $\{1,2,\dots,n\}$. For $0\le \alpha \le 1$, set $K=\sum_i X(i)^\alpha Y(i)^{1-\alpha}$ so that $Z:=\frac{1}{K}X^\alpha Y^{1-\alpha}$ is also a probability ...

**28**

votes

**1**answer

2k views

### An Entropy Inequality

Let $X,Y$ be probability measures on $\{1,2,\dots,n\}$, and set $K=\sum_i\sqrt{X(i)Y(i)}$ so that $Z:=\frac{1}{K}\sqrt{XY}$ is also a probability measure on $\{1,2,\dots,n\}$. How can we prove the ...

**4**

votes

**3**answers

409 views

### Incremental entropy computation

After a quick internet search I found no method for incremental entropy computation.
Question 1
Let $\{x_i\}_{i=1}^n$ and $\{x_i\}_{i=1+n}^{n+m}$ be two samples and let $S_i^j:=\sum_{k=i}^j x_k$. ...

**3**

votes

**0**answers

151 views

### Axioms for Mutual Information

I am interesting in axiomatic justifications for concepts in information theory. I have found many axiomatizations for Shannon's entropy and for the Kullback-Leibler divergence, as well as their ...

**2**

votes

**1**answer

257 views

### A Johnson-Lindenstrauss lemma for finite fields?

Given $m$ points in $\mathbb{R}^N$, the Johnson-Lindenstrauss lemma guarantees the existence of a linear operator $\mathbb{R}^N\rightarrow\mathbb{R}^n$ that nearly preserves pairwise distances between ...

**5**

votes

**0**answers

152 views

### Chernoff bound in the not-quite-sub-exponential case

In Terry Tao's notes on Concentration of measure, Exercise 7 indicates that the Chernoff bound can be generalized to sub-exponential random variables:
...

**2**

votes

**1**answer

162 views

### KL divergence(s) comparison,

Hi,
$P_1$, $P_2$, $P_3$ are probability distributions defined on the same support.
Knowing that $H(P_1) < H(P_2) < H(P_3)$, can we compare $D_{KL}(P_2,P_1)$ and $D_{KL}(P_3,P_1)$ ?
(H is the ...

**1**

vote

**1**answer

210 views

### Calculate channel capacity of general channel under constraint

Hi!
Given a conditional distribution $P_{Y|X}$ I'd like to find the prior distribution $P_X$ that maximizes the mutual information $I(X;Y)$ with $P_Y(y)=\int P_{Y|X}(y|x)P_X(x)\text{dx}$ (this ...

**1**

vote

**1**answer

238 views

### Derivative of a random process

Consider $w(t)$ as Guassian random process, with $w(t)$ being $\mathcal{N}(\mu,\sigma)$ and i.i.d for all t.
I consider applying a (stochastic)derivative operation to the random process. What is the ...

**7**

votes

**0**answers

201 views

### Uncertainty principle in Entropy terms

Math Questions:
Consider Hilbert space $L_2(\mathbb{R})$ with a standard norm
$
||\psi|| = ( \int_{\mathbb{R}}{ |\psi(t)|^2 dt } )^{1/2},
$
and Fourier transform
$
(F\psi)(\xi) =
...

**2**

votes

**1**answer

143 views

### Moments of random matrices - when are they finite

I need to evaluate the moment
$$\mathbb{E} (AX)^n,$$ where A is an NxN Hermitian square matrix, and X is
$$X=ZZ^{\ast},$$ where
$Z=\mu+Y$, where $\mu$ is mean of $Z$ and $Y$ is a zero-mean complex ...

**2**

votes

**0**answers

68 views

### “Soft” Voronoi cells or statistical criterias

It is probably some basic statistics question, but...
Informally 1: How to choose "criteria", such that it will guarantee that error decision probability is less than "epsilon", and maximize ...

**3**

votes

**0**answers

120 views

### What distribution(s) of delays make(s) timing attacks hardest?

$H$ is (Shannon) entropy.
In terms of the positive real number $t$, what distribution(s) $\hspace{.01 in}X$ on $\:[0\hspace{.005 in},\hspace{-0.03 in}\scriptsize+\normalsize\infty\hspace{-0.02 ...

**5**

votes

**2**answers

455 views

### Sufficient Condition for Exponential Decay in Chernoff Bound (Large Deviations)

Let $X_i$ ($i=1,...,n$) be a sequence of independent and identically distributed random variables. Denote $\mu=\mathbb{E}[X_i]$ and $S_n=\frac{1}{n}\sum_{i=1}^nX_i$. This question concerns the tail ...

**3**

votes

**1**answer

223 views

### convex combination of two covariance estimates

I am interested in covaraince matrix estimation. In brief: I have two estimates of the covariance matrix, and now I want to form a bona fide convex combination of the two.
Background: I have studied ...

**3**

votes

**0**answers

126 views

### Find a minimum entropy code for a simple gibbs random field.

Just to make precise what I am talking about, I will include the definition of a minimum entropy code. I will then define the precise markov random field I am asking about.
In the rest of this ...

**4**

votes

**2**answers

342 views

### Proving that a complicated function is eventually concave

I have a function $f:\mathbb{R}^+ \to \mathbb{R}^+$ that I want to prove is eventually concave - i.e. that there exists $\gamma _0 > 0$ such that for every $\gamma>\gamma_0$, $f(\gamma)$ is ...

**4**

votes

**2**answers

252 views

### Maximum entropy priors in infinite dimensional spaces

Is there an extension of maximum entropy probability distributions for function spaces?
For $\mathbb{R}^n$ and discrete spaces, there is much literature about this problem under names such as ...

**0**

votes

**1**answer

178 views

### MMSE estimator expressed through cumulants

I have a linear model $$Y=HX+N,$$ where $H$ is a matrix and $X$ are drawn from $p_X(X)$, and $N$ is Gaussian noise variates.
Now, if $X$ is multivariate Gaussian, then a linear estimator ...

**3**

votes

**1**answer

1k views

### With Huffman code, why do we still need Shannon code?

I'm studying information theory by myself.
I'm confused about that since we already have Huffman code, which is the optimal code method, why are Shannon code and some other code still useful?
I ...

**1**

vote

**1**answer

128 views

### The degrees in a random subgraph

Fix some positive integers $N$ and $d_k$, $k=1,2,\dots$ with $N=\sum_{k=1}^\infty d_k$.
Suppose you have a graph $G$ taken randomly uniformly among the set of all (unoriented) graphs with $N$ ...

**0**

votes

**2**answers

176 views

### multivariate distributions unaffected by unitary transformations

Hi,
In my research I reached some very nice results for IID complex Gaussian vectors $\bf{x}$.
Now I realize that my results hold for any random vectors that are unaffected by a unitary map, i.e., ...

**6**

votes

**2**answers

169 views

### Number of neigbour Voronoi cells for a random set of points on S^k or cube [-1, 1]^k?

Consider $S^k \subset R^{k+1} $. Sample $N$ points by say uniform distribution. (Example k=120, N=2^24, i.e. N>>k ).
Consider Voronoi cell around each point.
How many neighbours would a cell have ...

**1**

vote

**0**answers

52 views

### Information amount of fuzzy data transfer [closed]

Suppose we have binary channel from which we are able to receive zeroes and ones. We also know apriory probability $p$ of receiving "1". Then we can calculate information amount of each digit $q$ we ...

**2**

votes

**1**answer

203 views

### Error bounds for truncating a probability distribution based on the entropy?

Heuristic Background
Consider a set of states labeled $n=1,2,...$ in order of non-increasing probability $p(n)$.
The standard Shannon argument gives meaning to the entropy $S$ of $p$ in terms of the ...

**1**

vote

**2**answers

275 views

### Measuring the independence between the components of a stochastic process

In a context of blind source separation (e.g. you want to extract the voice of a singer from a song), many approaches consist in maximizing the independence between the components of a certain ...

**1**

vote

**1**answer

354 views

### Bounding Entropy in terms of KL-Divergence

Let $h(X)$ be the differential entropy of a continuous random variable $X$ with density $f$, and let $Y$ be another continuous random variable with density $g$. If $KL(X\mid\mid Y)$ is the ...

**3**

votes

**0**answers

91 views

### Is a parametric family which is universally consistent for multiple quantiles impossible?

Suppose I am dead-set on using Bayesian inference on independent and identically distributed data, but I'm lazy and insist on using a parametric likelihood function come what may. I'd be reassured to ...

**2**

votes

**1**answer

396 views

### Divergence between two random variables

I have two Gaussian random variables $X$ and $Y$, each of which is an estimator of an underlying quantity. I need to measure whether $Y$ is estimating something different than $X$. So if the mean of ...

**2**

votes

**0**answers

123 views

### finding rank-3 tensors compatible with a rank-2 tensor projection

I am interested in the following problem: Consider a rank-3 symmetric tensor $\boldsymbol{\sigma}$ with $\sigma_{ijk}$ where $\sigma_{ijk}$ can be 0 or 1, and the symmetry is with respect to any ...

**8**

votes

**2**answers

541 views

### Is this a situation where triple mutual information is always non-negative?

Suppose I have three identically-distributed homogeneous continuous-time discrete state space Markov chains $X_1(t), X_2(t), X_3(t)$, $t\geq 0$. They evolve independently but share a common random ...

**1**

vote

**0**answers

191 views

### Use of a priori information

I'm reading a paper [R1] where the authors propose a MAP estimator of the phase noise and frequency offset. However equation (17), which I reproduce below, represents a challenging step and I would ...

**2**

votes

**0**answers

346 views

### How to calculate/approximate expectation of function of a binomial random variable?

Hi,
I am stuck at following problem in my research.
Suppose that $M=m$ is a random variable with binomial distribution with parameters $n,p$. The constants $r$ and $\gamma$ are greater than zero. ...

**0**

votes

**0**answers

621 views

### Calculate entropy for a set of data

Hi;
I am not really a maths person and I have a question regarding shannon entropy. I have different datasets which only consists of three letter(I,N,M) such as :
dataset1: {I,I,N,M,I,I}
dataset2: ...

**2**

votes

**1**answer

324 views

### metric for signal to noise ratio in communication systems

I'm not quite sure about how to define a good measure of the quality of a communication channel with fading and interference. Let us assume the simplest case, where a node in a network receives the ...

**9**

votes

**3**answers

884 views

### Entropy of a measure

Let $\mu$ be a probability measure on a set of $n$ elements and let $p_i$ be the measure of the $i$-th element. Its Shannon entropy is defined by
$$
E(\mu)=-\sum_{i=1}^np_i\log(p_i)
$$
with the ...