The entropy tag has no wiki summary.

**29**

votes

**1**answer

2k views

### An Entropy Inequality

Let $X,Y$ be probability measures on $\{1,2,\dots,n\}$, and set $K=\sum_i\sqrt{X(i)Y(i)}$ so that $Z:=\frac{1}{K}\sqrt{XY}$ is also a probability measure on $\{1,2,\dots,n\}$. How can we prove the ...

**16**

votes

**1**answer

838 views

### An Entropy Inequality (generalized)

Let $X,Y$ be probability measures on $\{1,2,\dots,n\}$. For $0\le \alpha \le 1$, set $K=\sum_i X(i)^\alpha Y(i)^{1-\alpha}$ so that $Z:=\frac{1}{K}X^\alpha Y^{1-\alpha}$ is also a probability ...

**15**

votes

**1**answer

665 views

### Gini Coefficient and Renyi Entropy

Gini coefficient (aka Gini Index) is a quantity used in economics to describe income inequality. It is 0 for uniformly distributed income, and approaches 1 when all income is in hands of one ...

**11**

votes

**0**answers

145 views

### Entropy in elimination theory, or a brief remark by Gelfand-Kapranov-Zelevinsky

In the introduction to their book "Discriminants, resultants and multidimensional determinant", the authors state a very intriguing observation concerning the coefficients of monomials appearing in ...

**10**

votes

**2**answers

156 views

### Intrinsic significance of differential entropy

Many commentators (e.g. Jaynes, Rota) argue that the notion of "differential entropy" is problematic (as commonly defined by $ h(X) = \int ( \log\frac{1}{p(x)} ) p(x) \, dx $, where $X$ is a random ...

**10**

votes

**2**answers

1k views

### Proving a messy inequality

EDIT:
After much work I was able to reduce the inequality to a single variable function which I need to show is non-positive. That function is (for $0\leq p\leq\frac{1}{2}$)
$$\frac{p^2(\log(p))^2 - ...

**10**

votes

**1**answer

1k views

### System with invariant measure, but no ergodic measure.

Question
Examples of continuous transformations $T: X \to X$ such that the family of invariant probability measures $M(T)$ is NOT empty but there is no ergodic measure ($E(T) = \emptyset$).
Notice ...

**10**

votes

**3**answers

724 views

### Supremum amongst Kolmogorov-Sinai entropies: ergodic or just invariant measures.

Cases where
$sup_{\mu \in E(T)} h_\mu(T)
\neq
\sup_{\mu \in M(T)} h_\mu(T)$.
Background
For a topological space $X$,
let $T: X \to X$ be a continuous application.
Then, call the set of ...

**9**

votes

**2**answers

275 views

### Entropy for Haar measure on $O(n)$

Let $G$ be a locally compact group. A measure $\mu$ is the right-Haar measure on $G$ if for every $g\in G$ and $E\subseteq G$ Borel set $\mu(Eg)=\mu(E)$. It is known that every locally compact group ...

**9**

votes

**1**answer

542 views

### Entropy of edit distance

The edit or Levenshtein distance between two strings is the minimum number of single character insertions, deletions and substitutions to transform one string into another. If we take random binary ...

**8**

votes

**2**answers

334 views

### Inequality in information theory

I am reading the paper "chain independence and common information" (http://ttic.uchicago.edu/~yury/papers/independ.pdf). In this paper, an inequality is used several times (without proof) which looks ...

**8**

votes

**1**answer

1k views

### Entropy of the Ising model

Consider the standard Ising model on $[0,N]^2$ for $N$ large. By that I mean the square-lattice Ising model without external field, inside an $N$-by-$N$ square. What is its entropy for $N$ large? It ...

**7**

votes

**1**answer

792 views

### What's the maximum entropy probability distribution given bounds [a,b] and mean?

What is the continuous probability distribution that maximizes entropy, given only the bounds of the random variable [a,b] and the mean mu of the probability distribution?
For example:
if a=0, b=1, ...

**7**

votes

**2**answers

315 views

### Entropy conjecture for distributions over $\mathbb{Z}_n$

Suppose we have two independent random variables $X$ (with distribution $p_X$) and $Y$ (with distribution $p_Y$) which take values in the cyclic group $\mathbb{Z}_n$. Let $Z = X +Y$, where the ...

**6**

votes

**1**answer

721 views

### Entropy of first return map and suspension flows

There are some well know formulas of Abramov about derived systems.
Firstly let $(X,\mu,f)$ be a probability preserving system and $A\subset X$ is measurable such that $\bigcup_{n\ge0}f^nA=X$. Let ...

**6**

votes

**1**answer

237 views

### A geometric/topology notion of Typical Sequences? Power of typical sequences in multiuser channels?

The idea of Typical sequences(http://en.wikipedia.org/wiki/Typical_set) is a crucial concept in Shannon's proof of the Noisy channel coding theorem. Unfortunately the notion is not sufficient to ...

**5**

votes

**2**answers

270 views

### Entropy of nested compact invariant sets

Let $f$ be a homeomorphism on a compact metric space $X$. $K_1\supset K_2\supset\cdots \supset K$ are compact subsets of $X$ such that $f(K_n)=K_n$ and
$K=\bigcap K_n$. If $h(f, K_1)<\infty$, do we ...

**5**

votes

**1**answer

596 views

### Bounding the entropy of a convolution

Say we have a function $f:\mathbb{Z}_2^n \to \mathbb{R}$, such that $\sum _{x\in \mathbb{Z}_2^n} f(x)^2 = 1$ (so we can think of $\{ f(x)^2\} _{x\in \mathbb{Z}_2^n}$ as a distribution). It is natural ...

**5**

votes

**0**answers

171 views

### Maximizing Renyi entropy for a certain channel

The channel under consideration is $T = A + B$, where $A$ and $B$ take on values in $\{0, 1\}$ according to a probability mass function. Let (joint) random vector $(A_1, A_2,\ldots, A_n)$ be denoted ...

**4**

votes

**2**answers

508 views

### Is there a simple relation between the entropy of a matrix and its characteristic polynomial?

Assume $M$ is an invertible positive matrix of rank $N$. The Von Neumann entropy $H$ of a matrix $M$ with eigenvalues $\{ \lambda_n \}$ is
$H[M] = -\sum_{n=1}^N \lambda_n \ln \lambda_n$.
In ...

**4**

votes

**1**answer

396 views

### Low degree polynomial approximation for the entropy function

Let $X$ be a discrete random variable with possible values
$\{x_1,\ldots,x_n\}$, and let $p$ denote the probability mass function of
$X$. In addition, denote $p_i=p(x_i)$.
The entropy of $X$ is ...

**4**

votes

**3**answers

435 views

### Incremental entropy computation

After a quick internet search I found no method for incremental entropy computation.
Question 1
Let $\{x_i\}_{i=1}^n$ and $\{x_i\}_{i=1+n}^{n+m}$ be two samples and let $S_i^j:=\sum_{k=i}^j x_k$. ...

**4**

votes

**2**answers

256 views

### Maximum entropy priors in infinite dimensional spaces

Is there an extension of maximum entropy probability distributions for function spaces?
For $\mathbb{R}^n$ and discrete spaces, there is much literature about this problem under names such as ...

**4**

votes

**1**answer

291 views

### Upper bound on joint Renyi entropy

Renyi entropy of a random pair $(X,Y)$ with probability distribution $p_{X,Y}$ is defined by
\begin{equation}
H_\alpha(X,Y) = \frac{1}{1-\alpha}\log\sum_{x,y} p_{X,Y}(x,y)^\alpha.
\end{equation}
...

**3**

votes

**3**answers

706 views

### The relations between the Perelman's entropy functional and notions of entropy from statistical mechanics

I am looking for the relations and analogies between the Perelman's entropy functional,$\mathcal{W}(g,f,\tau)=\int_M [\tau(|\nabla f|^2+R)+f-n] (4\pi\tau)^{-\frac{n}{2}}e^{-f}dV$, and notions of ...

**3**

votes

**2**answers

143 views

### Estimate entropy of a binary process in terms of decay of correlations

Suppose $( X_{n} )$ is an ergodic binary process with
$$
\mathbb P(X_{n}=1)= \mathbb P(X_{n}=0)=\frac 12.
$$
Naturally the entropy (rate) $h(X)$ of $X=(X_{n})$ satisfies
$$
h(X)=\lim_{n\to\infty} ...

**3**

votes

**1**answer

366 views

### Simple reason that a mathematician cannot do better than random when guessing contents of a box?

I have a question about the finite analog of the puzzle proposed here involving mathematicians guessing the contents of boxes.
Specifically, suppose there are $k$ unopened boxes each containing a ...

**3**

votes

**1**answer

162 views

### Ising entropy of a finite L_1 x L_2 lattice

We know the entropy per site of the 2-d Ising model from Onsager's solution.
Has anybody also calculated the entropy for a finite rectangle of size L_1 x L_2
with periodic boundary conditions (i.e. on ...

**3**

votes

**2**answers

887 views

### Optimization of relative entropy

Wondering if my following question is an application of information theory:
Lets say we have a factory and ship boxes of stuff outside. If a competitor stands outside my factory, observes the stream ...

**3**

votes

**1**answer

124 views

### General additive function of probability

Let $H$ be a function of finite sequences of probabilities (non-negative numbers summing up to 1) into real numbers, such that:
$H$ is continuous,
$H$ is symmetric w.r.t. the order of its arguments,
...

**3**

votes

**1**answer

118 views

### Example of Girsanov change of density with finite relative entropy, but with infinite integral over squared changed drift

Let $(\Omega, (\mathcal F_t), \mathbb P)$ denote the usual Wiener space where $\Omega = C[0,\infty)$, etc., and where $(W_t)_{t \geq 0}$ denotes the Wiener process.
Let $Z \in L^1(\mathbb P)$ with $Z ...

**3**

votes

**1**answer

509 views

### Convergence of Markov chains in terms of relative entropy

Consider a finite state, irreducible Markov chain with a rate matrix $Q$ and a stationary distribution $\pi$. Suppose the chain starts with the initial distribution $p$ at time $0$, then at time $t$ ...

**3**

votes

**1**answer

367 views

### Infinitely Divisible Distributions and Maximal Entropy

The normal distribution on $\mathbb{R}$, the exponential distribution on $\mathbb{R}_{\geq 0}$, and the geometric distribution on $\mathbb{N}$ are examples of distributions that are both infinitely ...

**3**

votes

**0**answers

75 views

### Is there a known generalization of the Schmidt decomposition based on a maximal set of “locally orthogonal” vectors?

I came across the following unusual generalization of the Schmidt decomposition in my work, which I describe below. I would like to know if this structure has been studied before so I can read more ...

**3**

votes

**0**answers

588 views

### Entropy conjecture for flows

The entropy conjecture for diffeomorphisms (see for example this paper) asserts that for diffeomorphisms of manifolds, the log of the spectral radius of the actions of the diffeomorphism on the ...

**2**

votes

**1**answer

106 views

### Is there a one-dimensional subshift of positive entropy s, all of whose sub-subshifts also have entropy s?

A subshift is a subset $X$ of $A^\mathbb{N}$ or $A^\mathbb{Z}$ (with $A$ finite), such that $X$ is topologically closed and closed under the shift operation. The shift operation is defined by ...

**2**

votes

**2**answers

249 views

### probability measures with entropy equal to nonnegative number

Is it true that for a given nonnegative number, there exists a measure-theoretical entropy value (supremum of entropies of all partitions under a measure-preserving transformation) that equals this ...

**2**

votes

**1**answer

210 views

### Compact group extension of a zero entropy system.

Suppose $T: X \to X$ is a continuous map
and $\mu$ a $T$-ergodic probability measure over the
Borel sets of $X$.
Now, suppose $K \subset \mathrm{Hom}(X)$ is a compact group
of measure-preserving ...

**2**

votes

**0**answers

107 views

### On the volume entropy of negatively curved manifolds

Let $X$ be the universal cover of a closed negatively curved Riemannian manifold. Let $x_0\in X$ be a base point, $S$ be the unit sphere in $T_{x_0}X$ and $\exp:T_{x_0}X\rightarrow X$ be the ...

**2**

votes

**0**answers

26 views

### largest size for a randomness extractor

I am not so expert in theoretical computer science, so sorry if the question is trivial, i just could not find it in literature.
Suppose we have a source $X$ with min-entropy $\ell$, the randomness ...

**2**

votes

**0**answers

47 views

### Private Randomness extractor

Suppose we are given two random variables $X$ and $Y$ with fixed marginal and joint distribution. What is the maximum randomness that we can extract from $Y$ that is independent from $X$, that is, if ...

**2**

votes

**0**answers

103 views

### metric entropy for Lipschitz functions

Suppose $(X,d)$ is a metric space of unit diameer and let $F$ be the collection of all $1$-Lipschitz functions mapping $X$ to $[-1,1]$, equipped with the sup-norm $||\cdot||_\infty$.
I am interested ...

**2**

votes

**0**answers

119 views

### Reference for and Properties of the $alpha$-entropy

Let $T : X \to X$ be a continuous map on, say, a compact metric space $X$. Let $\mu$ be an invariant borel measure. Under suitable conditions, a result of Brin and Katok states that $\mu$-almost ...

**2**

votes

**0**answers

84 views

### order of convergence of the conditional entropy (3)

I'm sorry for having open two questions which have been solved by elementary counter-examples provided by @AnthonyQuas. Actually I'm not an expert in information theory and I expected that a positive ...

**2**

votes

**0**answers

108 views

### A.G. Vitushkin's “Easily representable families of functions” - can it be generalized?

Background
In his monograph "Estimation of the complexity of the tabulation problem" (translated into English as "Theory of the Transmission and Processing of Information") Vitushkin studies ...

**2**

votes

**0**answers

277 views

### Does this inequality of negative relative entropy and quantum relative entropy hold?

Hello, everyone!
Question
I have a question about the relationship between general relative entropy and general quantum relative entropy: Given a unit vector $|i\rangle$ and two Hermitian matrices ...

**1**

vote

**3**answers

2k views

### Entropy of Random Signal

How would you proof that for all equal variance (Continuous) Random Signals, a Gaussian one would have the largest Entropy. Or in other words, Given a Variance Gaussian PDF maximizes the Entropy.
...

**1**

vote

**3**answers

110 views

### How to estimate the entropy of a distribution on a power set?

Given a probability distribution $(X,p)$, its entropy is defined as $H=-\sum_{x\in X} p(x)\log p(x)$.
Given a sample of observations $x_n,n=1..N$, one can estimate $p(x)=\frac{\#\{i:x_i=x\}}{N}$ and ...

**1**

vote

**1**answer

4k views

### How to compute KL-divergence when PMF contains 0s?

From the Wikipedia page on Kullback-Leibler divergence, the way to compute this metric is to utilize the following formula:
The way I understand this is to compute the PMFs of two given sample sets ...

**1**

vote

**1**answer

93 views

### order of convergence of the conditional entropy (2)

Let $X_n$ be a random variable distributed on $A_n:=\{1, \ldots, n\}$ and $g_n\colon A_n \to A_n$ such that $\Pr\big(X_n \neq g_n(X_n)\big) \to 0$. Putting $Y_n=g_n(X_n)$, then by Fano's inequality ...