The entropy tag has no usage guidance.

**30**

votes

**2**answers

2k views

### An Entropy Inequality

Let $X,Y$ be probability measures on $\{1,2,\dots,n\}$, and set $K=\sum_i\sqrt{X(i)Y(i)}$ so that $Z:=\frac{1}{K}\sqrt{XY}$ is also a probability measure on $\{1,2,\dots,n\}$. How can we prove the ...

**18**

votes

**2**answers

1k views

### An Entropy Inequality (generalized)

Let $X,Y$ be probability measures on $\{1,2,\dots,n\}$. For $0\le \alpha \le 1$, set $K=\sum_i X(i)^\alpha Y(i)^{1-\alpha}$ so that $Z:=\frac{1}{K}X^\alpha Y^{1-\alpha}$ is also a probability ...

**17**

votes

**8**answers

2k views

### When do people actually use the maximum entropy distribution?

One of the standard problems in convex optimization is the calculation of the maximum entropy distribution that satisfies some set of criteria. For example, if $\mathbf{x} \in \mathbb R^n$ is an ...

**16**

votes

**1**answer

1k views

### Gini Coefficient and Renyi Entropy

Gini coefficient (aka Gini Index) is a quantity used in economics to describe income inequality. It is 0 for uniformly distributed income, and approaches 1 when all income is in hands of one ...

**15**

votes

**3**answers

521 views

### Intrinsic significance of differential entropy

Many commentators (e.g. Jaynes, Rota) argue that the notion of "differential entropy" is problematic (as commonly defined by $ h(X) = \int ( \log\frac{1}{p(x)} ) p(x) \, dx $, where $X$ is a random ...

**12**

votes

**0**answers

189 views

### Entropy in elimination theory, or a brief remark by Gelfand-Kapranov-Zelevinsky

In the introduction to their book "Discriminants, resultants and multidimensional determinant", the authors state a very intriguing observation concerning the coefficients of monomials appearing in ...

**11**

votes

**1**answer

668 views

### Entropy of edit distance

The edit or Levenshtein distance between two strings is the minimum number of single character insertions, deletions and substitutions to transform one string into another. If we take random binary ...

**11**

votes

**1**answer

1k views

### System with invariant measure, but no ergodic measure.

Question
Examples of continuous transformations $T: X \to X$ such that the family of invariant probability measures $M(T)$ is NOT empty but there is no ergodic measure ($E(T) = \emptyset$).
Notice ...

**10**

votes

**2**answers

1k views

### Proving a messy inequality

EDIT:
After much work I was able to reduce the inequality to a single variable function which I need to show is non-positive. That function is (for $0\leq p\leq\frac{1}{2}$)
$$\frac{p^2(\log(p))^2 - ...

**10**

votes

**3**answers

793 views

### Supremum amongst Kolmogorov-Sinai entropies: ergodic or just invariant measures.

Cases where
$sup_{\mu \in E(T)} h_\mu(T)
\neq
\sup_{\mu \in M(T)} h_\mu(T)$.
Background
For a topological space $X$,
let $T: X \to X$ be a continuous application.
Then, call the set of ...

**9**

votes

**2**answers

403 views

### Entropy for Haar measure on $O(n)$

Let $G$ be a locally compact group. A measure $\mu$ is the right-Haar measure on $G$ if for every $g\in G$ and $E\subseteq G$ Borel set $\mu(Eg)=\mu(E)$. It is known that every locally compact group ...

**9**

votes

**1**answer

1k views

### What's the maximum entropy probability distribution given bounds [a,b] and mean?

What is the continuous probability distribution that maximizes entropy, given only the bounds of the random variable [a,b] and the mean mu of the probability distribution?
For example:
if a=0, b=1, ...

**9**

votes

**2**answers

425 views

### Expected centered entropy of the binomial distribution

In short, the function I am interested in is the following:
$$I_n(p) = \sum_{k=0}^n \binom{n}{k} p^k (1-p)^{n-k} \left[h(p) - h\left(\tfrac{k}{n}\right)\right],$$
where $h(x) \triangleq -x \log x - ...

**8**

votes

**2**answers

436 views

### Inequality in information theory

I am reading the paper "chain independence and common information" (http://ttic.uchicago.edu/~yury/papers/independ.pdf). In this paper, an inequality is used several times (without proof) which looks ...

**8**

votes

**1**answer

1k views

### Entropy of first return map and suspension flows

There are some well know formulas of Abramov about derived systems.
Firstly let $(X,\mu,f)$ be a probability preserving system and $A\subset X$ is measurable such that $\bigcup_{n\ge0}f^nA=X$. Let ...

**8**

votes

**1**answer

2k views

### Entropy of the Ising model

Consider the standard Ising model on $[0,N]^2$ for $N$ large. By that I mean the square-lattice Ising model without external field, inside an $N$-by-$N$ square. What is its entropy for $N$ large? It ...

**8**

votes

**1**answer

222 views

### Higher moments of information and Renyi entropy

For a given discrete probability distribution, Shannon entropy can be though as an expectation value $\langle - \log p \rangle$ (see also: What is entropy, really?, What is the role of the logarithm ...

**8**

votes

**2**answers

363 views

### Entropy conjecture for distributions over $\mathbb{Z}_n$

Suppose we have two independent random variables $X$ (with distribution $p_X$) and $Y$ (with distribution $p_Y$) which take values in the cyclic group $\mathbb{Z}_n$. Let $Z = X +Y$, where the ...

**7**

votes

**1**answer

6k views

### How to compute KL-divergence when PMF contains 0s?

From the Wikipedia page on Kullback-Leibler divergence, the way to compute this metric is to utilize the following formula:
The way I understand this is to compute the PMFs of two given sample sets ...

**7**

votes

**2**answers

209 views

### Estimating entropy conditional to an event

Take for example the measure $\mu(n)=n^2$ on $\{1, \ldots, N\}$ and a random variable $X$ distributed according to the probability obtained by normalizing $\mu$.
Does there exists a constant ...

**7**

votes

**1**answer

213 views

### Maximal entropy distribution with given conditionals

It is well known that of all the joint distributions $p(x,y)$ with fixed marginals $p(x),p(y)$, the one with the highest entropy is:
$$
p(x,y)=p(x)p(y).
$$
Suppose instead that we have conditionals. ...

**6**

votes

**2**answers

284 views

### Entropy of nested compact invariant sets

Let $f$ be a homeomorphism on a compact metric space $X$. $K_1\supset K_2\supset\cdots \supset K$ are compact subsets of $X$ such that $f(K_n)=K_n$ and
$K=\bigcap K_n$. If $h(f, K_1)<\infty$, do we ...

**6**

votes

**1**answer

248 views

### A geometric/topology notion of Typical Sequences? Power of typical sequences in multiuser channels?

The idea of Typical sequences(http://en.wikipedia.org/wiki/Typical_set) is a crucial concept in Shannon's proof of the Noisy channel coding theorem. Unfortunately the notion is not sufficient to ...

**6**

votes

**1**answer

219 views

### Finding discrete entropy via differential entropy

In a recent math.se question the following was asked which I have slightly edited.
"
Consider a fixed and given $n$ by $n$ matrix $M$ whose elements are chosen from $\{-1,1\}$. Consider also a random ...

**5**

votes

**2**answers

221 views

### General additive function of probability

Let $H$ be a function of finite sequences of probabilities (non-negative numbers summing up to 1) into real numbers, such that:
$H$ is continuous,
$H$ is symmetric w.r.t. the order of its arguments,
...

**5**

votes

**1**answer

759 views

### Bounding the entropy of a convolution

Say we have a function $f:\mathbb{Z}_2^n \to \mathbb{R}$, such that $\sum _{x\in \mathbb{Z}_2^n} f(x)^2 = 1$ (so we can think of $\{ f(x)^2\} _{x\in \mathbb{Z}_2^n}$ as a distribution). It is natural ...

**5**

votes

**0**answers

42 views

### Basis functions for approximation of a convex function on unit simplex

Consider the unit $D$-simplex $S^D=\left\lbrace (x_0, x_1, \ldots, x_D) \in \mathbb{R}^{D+1} \mid \sum\limits_{i=0}^{D}x_i = 1, x_i \geq 0 \right\rbrace$. I have a bounded, convex function ...

**5**

votes

**0**answers

222 views

### Maximizing Renyi entropy for a certain channel

The channel under consideration is $T = A + B$, where $A$ and $B$ take on values in $\{0, 1\}$ according to a probability mass function. Let (joint) random vector $(A_1, A_2,\ldots, A_n)$ be denoted ...

**4**

votes

**2**answers

824 views

### Is there a simple relation between the entropy of a matrix and its characteristic polynomial?

Assume $M$ is an invertible positive matrix of rank $N$. The Von Neumann entropy $H$ of a matrix $M$ with eigenvalues $\{ \lambda_n \}$ is
$H[M] = -\sum_{n=1}^N \lambda_n \ln \lambda_n$.
In ...

**4**

votes

**1**answer

541 views

### Low degree polynomial approximation for the entropy function

Let $X$ be a discrete random variable with possible values
$\{x_1,\ldots,x_n\}$, and let $p$ denote the probability mass function of
$X$. In addition, denote $p_i=p(x_i)$.
The entropy of $X$ is ...

**4**

votes

**3**answers

565 views

### Incremental entropy computation

After a quick internet search I found no method for incremental entropy computation.
Question 1
Let $\{x_i\}_{i=1}^n$ and $\{x_i\}_{i=1+n}^{n+m}$ be two samples and let $S_i^j:=\sum_{k=i}^j x_k$. ...

**4**

votes

**1**answer

414 views

### Infinitely Divisible Distributions and Maximal Entropy

The normal distribution on $\mathbb{R}$, the exponential distribution on $\mathbb{R}_{\geq 0}$, and the geometric distribution on $\mathbb{N}$ are examples of distributions that are both infinitely ...

**4**

votes

**2**answers

289 views

### Maximum entropy priors in infinite dimensional spaces

Is there an extension of maximum entropy probability distributions for function spaces?
For $\mathbb{R}^n$ and discrete spaces, there is much literature about this problem under names such as ...

**4**

votes

**1**answer

410 views

### Upper bound on joint Renyi entropy

Renyi entropy of a random pair $(X,Y)$ with probability distribution $p_{X,Y}$ is defined by
\begin{equation}
H_\alpha(X,Y) = \frac{1}{1-\alpha}\log\sum_{x,y} p_{X,Y}(x,y)^\alpha.
\end{equation}
...

**4**

votes

**0**answers

98 views

### $q$-deformations of fundamental equation of information and entropies

Classical information theory: fundamental equation of information
In classical information theory, the information $I(A)$ of an event $A$ (any element of the $\sigma$-algebra $\mathcal F$ of a ...

**4**

votes

**0**answers

668 views

### Entropy conjecture for flows

The entropy conjecture for diffeomorphisms (see for example this paper) asserts that for diffeomorphisms of manifolds, the log of the spectral radius of the actions of the diffeomorphism on the ...

**3**

votes

**3**answers

1k views

### The relations between the Perelman's entropy functional and notions of entropy from statistical mechanics

I am looking for the relations and analogies between the Perelman's entropy functional,$\mathcal{W}(g,f,\tau)=\int_M [\tau(|\nabla f|^2+R)+f-n] (4\pi\tau)^{-\frac{n}{2}}e^{-f}dV$, and notions of ...

**3**

votes

**1**answer

190 views

### Is there a one-dimensional subshift of positive entropy s, all of whose sub-subshifts also have entropy s?

A subshift is a subset $X$ of $A^\mathbb{N}$ or $A^\mathbb{Z}$ (with $A$ finite), such that $X$ is topologically closed and closed under the shift operation. The shift operation is defined by ...

**3**

votes

**2**answers

172 views

### Estimate entropy of a binary process in terms of decay of correlations

Suppose $( X_{n} )$ is an ergodic binary process with
$$
\mathbb P(X_{n}=1)= \mathbb P(X_{n}=0)=\frac 12.
$$
Naturally the entropy (rate) $h(X)$ of $X=(X_{n})$ satisfies
$$
h(X)=\lim_{n\to\infty} ...

**3**

votes

**2**answers

183 views

### Maximizing entropy under constraints

This question is about an extension of the variational principle in thermodynamical formalism when one adds linear constraints to the measures.
Consider the one-sided shift ...

**3**

votes

**1**answer

404 views

### Simple reason that a mathematician cannot do better than random when guessing contents of a box?

I have a question about the finite analog of the puzzle proposed here involving mathematicians guessing the contents of boxes.
Specifically, suppose there are $k$ unopened boxes each containing a ...

**3**

votes

**1**answer

772 views

### Convergence of Markov chains in terms of relative entropy

Consider a finite state, irreducible Markov chain with a rate matrix $Q$ and a stationary distribution $\pi$. Suppose the chain starts with the initial distribution $p$ at time $0$, then at time $t$ ...

**3**

votes

**1**answer

182 views

### Ising entropy of a finite L_1 x L_2 lattice

We know the entropy per site of the 2-d Ising model from Onsager's solution.
Has anybody also calculated the entropy for a finite rectangle of size L_1 x L_2
with periodic boundary conditions (i.e. on ...

**3**

votes

**2**answers

1k views

### Optimization of relative entropy

Wondering if my following question is an application of information theory:
Lets say we have a factory and ship boxes of stuff outside. If a competitor stands outside my factory, observes the stream ...

**3**

votes

**1**answer

236 views

### Information theory from negative probability

Szekely provides a convincing argument of negative probability here:
http://www.wilmott.com/pdfs/100609_gjs.pdf
What does a reformulation of classical information theory built from negative ...

**3**

votes

**1**answer

149 views

### Example of Girsanov change of density with finite relative entropy, but with infinite integral over squared changed drift

Let $(\Omega, (\mathcal F_t), \mathbb P)$ denote the usual Wiener space where $\Omega = C[0,\infty)$, etc., and where $(W_t)_{t \geq 0}$ denotes the Wiener process.
Let $Z \in L^1(\mathbb P)$ with $Z ...

**3**

votes

**2**answers

244 views

### Entropy equals zero?

Imagine you have a shift invariant ($\sigma$-invariant) probability measure $\eta$
in the Bernoulli space $\{0,1\}^{\mathbb{N}}$. Define
$\mathcal{P} = \{[0],[1]\}$;
$\mathcal{P}^{n} = ...

**3**

votes

**1**answer

244 views

### Convolution of measures - entropy growth

Imagine you have two shift-invariant measures $\mu, \nu$ in the Bernoulli space $\{0,1\}^{\mathbb{N}}$ with positive entropy and both are not the Bernoulli measure $(\frac{1}{2},\frac{1}{2})$. I know ...

**3**

votes

**1**answer

124 views

### Average entropy of quantum system in bipartite pure state for finite temperature

[I got halfway through writing this when I found the paper that answers the question in (essentially) the affirmative. I'll post it anyways in case anyone is interested.]
Background: If a random ...

**3**

votes

**1**answer

246 views

### Compact group extension of a zero entropy system.

Suppose $T: X \to X$ is a continuous map
and $\mu$ a $T$-ergodic probability measure over the
Borel sets of $X$.
Now, suppose $K \subset \mathrm{Hom}(X)$ is a compact group
of measure-preserving ...