The entropy tag has no usage guidance.

**4**

votes

**1**answer

229 views

### Do binary symmetric channels maximize mutual information?

Consider the following setup: $(X, Y)$ is a doubly symmetric binary source with parameter $0 < p < 1/2$, i.e., $X \sim \text{Bernoulli}(1/2)$, $Z \sim \text{Bernoulli}(p)$ and $Y = X \oplus Z$. ...

**1**

vote

**0**answers

60 views

### How to prove this Gaussian Mixture theorem? (Fitting/Overfitting)

In certain applications, we approximate an unknown pdf by placing uniformly weighted Gaussian terms at each of some sample points $\{x_{1},...,x_{n}\}
$ and assigning some variances $\{\sigma_{1},...,...

**3**

votes

**3**answers

224 views

### Maximizing entropy under constraints

This question is about an extension of the variational principle in thermodynamical formalism when one adds linear constraints to the measures.
Consider the one-sided shift $\sigma:\mathcal{A}^\...

**1**

vote

**2**answers

53 views

### Reference for the monotonicity in $\alpha$ of the Rényi entropy

I'd like to have a reference for the property $0 \leq \alpha < \alpha' \leq \infty \implies R_\alpha(\mu) > R_{\alpha'}(\mu)$, where $R_\alpha(\mu)$ is the Rényi entropy of order $\alpha$ of a ...

**7**

votes

**2**answers

326 views

### Maximal entropy distribution with given conditionals

It is well known that of all the joint distributions $p(x,y)$ with fixed marginals $p(x),p(y)$, the one with the highest entropy is:
$$
p(x,y)=p(x)p(y).
$$
Suppose instead that we have conditionals. ...

**1**

vote

**1**answer

42 views

### Maximizing joint entropy?

I'm stuck trying to find the maximum entropy probability distribution taking into account a joint distribution.
Basically, I want to find the maximum entropy expression for $p(x,y)$ when the marginal ...

**2**

votes

**1**answer

543 views

### Calculate channel capacity of general channel under constraint

Given a conditional distribution $P_{Y|X}$ I'd like to find the prior distribution $P_X$ that maximizes the mutual information $I(X;Y)$ with $P_Y(y)=\int P_{Y|X}(y|x)P_X(x)\text{dx}$ (this corresponds ...

**-1**

votes

**1**answer

85 views

### Proving maximal entropy [closed]

It is quite easy to prove that
$$H(S) \leq \log_2(|A|),$$
where $A$ is the number of events, using the Jensen inequality
$$H(S) = E_S[\log_2(\frac{1}{P_S(s)})]\leq \log_2(E_S[(\frac{1}{P_S(s)})]) =...

**2**

votes

**1**answer

123 views

### About Renyi entropy

If one is given a joint probability distribution over a finite set of discrete random variables then I guess there a notion of $\alpha-$Renyi entropy defined for it as $S_\alpha (X_1,..,X_n) = \frac{...

**3**

votes

**0**answers

78 views

### An inequality involving conditional variance and its connection to information theory

Given absolutely continuous random variables $(X, Y)$ with joint distribution $P_{XY}$, we construct $Z:=\sqrt{\gamma} Y+N_\mathsf{G}$ where $N_\mathsf{G}\sim N(0, 1)$ and is independent of $(X,Y)$ ...

**7**

votes

**1**answer

817 views

### Bounding the entropy of a convolution

Say we have a function $f:\mathbb{Z}_2^n \to \mathbb{R}$, such that $\sum _{x\in \mathbb{Z}_2^n} f(x)^2 = 1$ (so we can think of $\{ f(x)^2\} _{x\in \mathbb{Z}_2^n}$ as a distribution). It is natural ...

**2**

votes

**0**answers

37 views

### TAP expression for entropy [closed]

This paper by Barton and Cocco: http://www.phys.ens.fr/~cocco/Art/articlejstat.pdf
claims on page 17 (Formula (30)) an expression for the "high-temperature" entropy of an Ising model, given its ...

**1**

vote

**0**answers

97 views

### weak-* versus entropy growth

General question. Let $\eta_{n}$ be a sequence of invariant measures on $\{0,1,2,...,p-1\}^{\mathbb{N}}$ and $B$ the Bernoulli uniform measure. Knowing that $\eta_{n} \rightarrow B$ in the weak-* ...

**30**

votes

**2**answers

2k views

### An Entropy Inequality

Let $X,Y$ be probability measures on $\{1,2,\dots,n\}$, and set $K=\sum_i\sqrt{X(i)Y(i)}$ so that $Z:=\frac{1}{K}\sqrt{XY}$ is also a probability measure on $\{1,2,\dots,n\}$. How can we prove the ...

**6**

votes

**1**answer

235 views

### Finding discrete entropy via differential entropy

In a recent math.se question the following was asked which I have slightly edited.
"
Consider a fixed and given $n$ by $n$ matrix $M$ whose elements are chosen from $\{-1,1\}$. Consider also a random ...

**1**

vote

**0**answers

94 views

### Strict factor of a dynamical system with the same entropy [closed]

Say that a factor of an invertible measure-preserving transformation $T$ is strict if it is not isomorphic to $T$. Does there exist an invertibe mpt $T$ such that $0 < h(T) < \infty$ and ...

**7**

votes

**2**answers

217 views

### Estimating entropy conditional to an event

Take for example the measure $\mu(n)=n^2$ on $\{1, \ldots, N\}$ and a random variable $X$ distributed according to the probability obtained by normalizing $\mu$.
Does there exists a constant $K>0$...

**0**

votes

**2**answers

7k views

### How to ensure the non-negativity of Kullback-Leibler Divergence KLD Metric (Relative Entropy)?

Hello,
I’m having some problems in ensuring the non-negativity of KLD!
I know that KLD is always positive and I went over the proof. However, it doesn’t seem to work for me. In some cases I’m ...

**3**

votes

**1**answer

257 views

### Convolution of measures - entropy growth

Imagine you have two shift-invariant measures $\mu, \nu$ in the Bernoulli space $\{0,1\}^{\mathbb{N}}$ with positive entropy and both are not the Bernoulli measure $(\frac{1}{2},\frac{1}{2})$. I know ...

**5**

votes

**0**answers

65 views

### Basis functions for approximation of a convex function on unit simplex

Consider the unit $D$-simplex $S^D=\left\lbrace (x_0, x_1, \ldots, x_D) \in \mathbb{R}^{D+1} \mid \sum\limits_{i=0}^{D}x_i = 1, x_i \geq 0 \right\rbrace$. I have a bounded, convex function $f:S^D\to\...

**0**

votes

**1**answer

114 views

### entropy growth of invariant measures - General question

In general, given a sequence of shift-invariant measures $\eta_{n}$ on $\{0,1\}^{\mathbb{N}}$ what to do to guarantee this convergence of entropies: $$h(\eta_{n}) \rightarrow \log2?$$
Because I'm ...

**1**

vote

**1**answer

118 views

### entropy and d-bar: how do we estimate continuity?

Let $G = \{0,1\}^{\mathbb{N}} = \mathbb{Z}_{2}^{\mathbb{N}}$ be the Bernoulli space of two symbols, let $\sigma$ be the shift map and $M(G)$ the set of $\sigma$-invariant probabilities. Let $\bar{d}$ ...

**4**

votes

**2**answers

261 views

### Entropy equals zero?

Imagine you have a shift invariant ($\sigma$-invariant) probability measure $\eta$
in the Bernoulli space $\{0,1\}^{\mathbb{N}}$. Define
$\mathcal{P} = \{[0],[1]\}$;
$\mathcal{P}^{n} = \mathcal{P}\...

**4**

votes

**1**answer

422 views

### Infinitely Divisible Distributions and Maximal Entropy

The normal distribution on $\mathbb{R}$, the exponential distribution on $\mathbb{R}_{\geq 0}$, and the geometric distribution on $\mathbb{N}$ are examples of distributions that are both infinitely ...

**0**

votes

**1**answer

146 views

### joining or coupling

given two shift invariant measures in the Bernoulli space $\{0,1\}^{\mathbb{N}}$, is there a way to construct joinings of them? It's very diffcult, in general, to find exactly the minimal joining i.e, ...

**5**

votes

**2**answers

226 views

### General additive function of probability

Let $H$ be a function of finite sequences of probabilities (non-negative numbers summing up to 1) into real numbers, such that:
$H$ is continuous,
$H$ is symmetric w.r.t. the order of its arguments,
...

**0**

votes

**0**answers

90 views

### Entropy, convergence and invariant measures

Could you give conditions that a sequence of shift invariant measures $\eta_{n}$ has to satisfy in order to happen this convergence in terms of entropies $h(\eta_{n})$:
$h(\eta_{n}) \rightarrow \log2$...

**11**

votes

**1**answer

1k views

### System with invariant measure, but no ergodic measure.

Question
Examples of continuous transformations $T: X \to X$ such that the family of invariant probability measures $M(T)$ is NOT empty but there is no ergodic measure ($E(T) = \emptyset$).
Notice ...

**1**

vote

**1**answer

173 views

### Entropy, Convergence

imagine you have a sequence $\eta_{n}$ of (shift) invariant measures in the Bernoulli space $\{0,1\}^{\mathbb{N}}$ that satisfy the following: there are a $0<\delta <1$ and an $N$ such that $$n &...

**3**

votes

**1**answer

266 views

### Information theory from negative probability

Szekely provides a convincing argument of negative probability here:
http://www.wilmott.com/pdfs/100609_gjs.pdf
What does a reformulation of classical information theory built from negative ...

**2**

votes

**2**answers

273 views

### Entropy inequality

Let $P,Q$ be probabilities on a finite set $A$ with $Q(a)\gt 0$, for all $a\in A$, and let $H(P)$, $H(Q)$ denote their entropies and $D(P\,\|\,Q)$ denote their Kullback-Leibler distance.
Is it always ...

**0**

votes

**1**answer

83 views

### Asymptotically full stationary process

Let $(X_n)_{n \in \mathbb{Z}}$ be a stationary process on a finite set $A$. Say that it is asymptotically full if for every increasing sequence of subsets $B_n \subset A^n$ such that $\dfrac{\#B_n}{\#...

**2**

votes

**2**answers

600 views

### Convexity and semicontinuity of the relative entropy function

There are several different definitions of relative entropy, and some of them are not equivalent. Following is the definition we will use in this question.
Let $M$ be a closed manifold and $\mathcal{...

**1**

vote

**0**answers

114 views

### Estimates of entropy of functional spaces

Let $M^n$ be a compact $n$-dimensional manifold. For $k\geq 0$ let us denote by $C^k(M)$ the Banach space of $k$ times continuously differentiable functions, and $B_{C^k}$ denote the unit ball of it.
...

**2**

votes

**1**answer

198 views

### Embeddings of subshifts

Consider $(X,\sigma_X)$ and $(Y, \sigma_{Y})$ be subshifts of the one sided shift in two symbols. Assume that $(X,\sigma_X)$ is a transitive subshift of finite type and $(Y, \sigma_{Y})$ is a ...

**9**

votes

**2**answers

480 views

### Expected centered entropy of the binomial distribution

In short, the function I am interested in is the following:
$$I_n(p) = \sum_{k=0}^n \binom{n}{k} p^k (1-p)^{n-k} \left[h(p) - h\left(\tfrac{k}{n}\right)\right],$$
where $h(x) \triangleq -x \log x - (1-...

**1**

vote

**1**answer

372 views

### Coupon Collector Problem for Non-Uniform Coupons: Bound on the number of missed Coupons

Suppose $\mathcal B=\{1,2,..,b\}$ is the set of all possible coupons, with $\mathbf p = ( p_1,p_2,...,p_b)$ assigning the probability of occurrence for all coupons in $\mathcal B$.
The "traditional ...

**2**

votes

**0**answers

96 views

### Concentration bound in high min entropy distribution

Let $(X_{1},\dots,X_{m})$ be joint distribution on $\{0,1\}^{m}$ with that $H_{\infty}(X_{1},\cdots,X_{m})\geq m-r$, where $H_{\infty}$ means min-entropy.
Let $P_{1},...,P_{n}\subseteq [m]$ be sets ...

**17**

votes

**8**answers

2k views

### When do people actually use the maximum entropy distribution?

One of the standard problems in convex optimization is the calculation of the maximum entropy distribution that satisfies some set of criteria. For example, if $\mathbf{x} \in \mathbb R^n$ is an ...

**16**

votes

**3**answers

612 views

### Intrinsic significance of differential entropy

Many commentators (e.g. Jaynes, Rota) argue that the notion of "differential entropy" is problematic (as commonly defined by $ h(X) = \int ( \log\frac{1}{p(x)} ) p(x) \, dx $, where $X$ is a random ...

**1**

vote

**1**answer

436 views

### Discrete Maximum Entropy Distribution with given mean

For a given mean $\mu$, what is the entropy maximizing probability distribution on the nonnegative integers?
Different sources indicated either the geometric or the Poisson distribution for this. As ...

**8**

votes

**2**answers

457 views

### Inequality in information theory

I am reading the paper "chain independence and common information" (http://ttic.uchicago.edu/~yury/papers/independ.pdf). In this paper, an inequality is used several times (without proof) which looks ...

**2**

votes

**0**answers

245 views

### Interpretation of Shannon Entropy Application

Consider a collection of increasing positive integers $\{a_i\}_{i=1}^m$ and the distribution $p_i=\frac{a_i}{\sum_{i=1}^ma_i}$. Let entropy of $\mathcal{A}=\{a_i\}_{i=1}^m$ be given
by $$H(\mathcal{A}...

**2**

votes

**0**answers

71 views

### Sequence transformations that are entropy invariant

Consider a collection of increasing positive integers $\{a_i\}_{i=1}^m$ and the distribution $p_i=\frac{a_i}{\sum_{i=1}^ma_i}$. Define entropy of $\mathcal{A}=\{a_i\}_{i=1}^m$
by $$H(\mathcal{A},m)=-\...

**1**

vote

**1**answer

111 views

### Limiting Entropy of deterministic sequences - 1

Consider a collection of increasing positive integers $\{a_i\}_{i=1}^m$.
Consider distribution $p_i=\frac{a_i}{\sum_{i=1}^ma_i}$. Given $\{a_i\}_{i=1}^m$, let $\mathcal{P}_{a,m}$ be distribution at ...

**1**

vote

**0**answers

44 views

### Limiting Entropy of deterministic sequences - 2

Consider a collection of increasing positive integers $\{a_i\}_{i=1}^m$ and the distribution $p_i=\frac{a_i}{\sum_{i=1}^ma_i}$. Given $\{a_i\}_{i=1}^m$, let $\mathcal{P}_a$ be limiting distribution $m\...

**1**

vote

**1**answer

86 views

### Entropy difference dominance of sequences

Consider a collection of positive integers $\{a_i\}_{i=1}^m$ and the distribution $p_i=\frac{a_i}{\sum_{i=1}^ma_i}$.
Similarly for the collection $\{a_i\}_{i=1}^{m+1}$ form the distribution $q_i=\...

**1**

vote

**1**answer

59 views

### Entropy dominance of certain restricted sequenes

Say you have positive $\{a_i\}_{i=1}^n$ and you have $p_i=\frac{a_i}{\sum_{i=1}^na_i}$, then assume you have a $C$ such that $C<2a_n\ll\sum_{i=1}^na_i$ (that is $C$ is not very large), then define $...

**2**

votes

**1**answer

81 views

### Entropy dominance

Let $0<a<b<c$ be distinct positive reals.
Define four different probability distributions:
$$\mathcal{P}_{ab}:P_{a,ab}=\frac{a}{a+b}=1-P_{b,ab}$$
$$\mathcal{P}_{bc}:P_{b,bc}=\frac{b}{b+c}=1-...

**8**

votes

**1**answer

235 views

### Higher moments of information and Renyi entropy

For a given discrete probability distribution, Shannon entropy can be though as an expectation value $\langle - \log p \rangle$ (see also: What is entropy, really?, What is the role of the logarithm ...