The entropy tag has no wiki summary.

**18**

votes

**2**answers

1k views

### An Entropy Inequality (generalized)

Let $X,Y$ be probability measures on $\{1,2,\dots,n\}$. For $0\le \alpha \le 1$, set $K=\sum_i X(i)^\alpha Y(i)^{1-\alpha}$ so that $Z:=\frac{1}{K}X^\alpha Y^{1-\alpha}$ is also a probability ...

**30**

votes

**1**answer

2k views

### An Entropy Inequality

Let $X,Y$ be probability measures on $\{1,2,\dots,n\}$, and set $K=\sum_i\sqrt{X(i)Y(i)}$ so that $Z:=\frac{1}{K}\sqrt{XY}$ is also a probability measure on $\{1,2,\dots,n\}$. How can we prove the ...

**4**

votes

**3**answers

503 views

### Incremental entropy computation

After a quick internet search I found no method for incremental entropy computation.
Question 1
Let $\{x_i\}_{i=1}^n$ and $\{x_i\}_{i=1+n}^{n+m}$ be two samples and let $S_i^j:=\sum_{k=i}^j x_k$. ...

**3**

votes

**3**answers

1k views

### The relations between the Perelman's entropy functional and notions of entropy from statistical mechanics

I am looking for the relations and analogies between the Perelman's entropy functional,$\mathcal{W}(g,f,\tau)=\int_M [\tau(|\nabla f|^2+R)+f-n] (4\pi\tau)^{-\frac{n}{2}}e^{-f}dV$, and notions of ...

**0**

votes

**1**answer

112 views

### order of convergence of the conditional entropy

Let $X_n$ be a random variable distributed on $A_n:=\{1, \ldots, n\}$ and $g_n\colon A_n \to A_n$ such that $\Pr\big(X_n \neq g_n(X_n)\big) \to 0$. Putting $Y_n=g(X_n)$ then by Fano's inequality ...

**9**

votes

**1**answer

618 views

### Entropy of edit distance

The edit or Levenshtein distance between two strings is the minimum number of single character insertions, deletions and substitutions to transform one string into another. If we take random binary ...

**2**

votes

**2**answers

411 views

### Convexity and semicontinuity of the relative entropy function

There are several different definitions of relative entropy, and some of them are not equivalent. Following is the definition we will use in this question.
Let $M$ be a closed manifold and ...

**1**

vote

**1**answer

403 views

### Calculate channel capacity of general channel under constraint

Hi!
Given a conditional distribution $P_{Y|X}$ I'd like to find the prior distribution $P_X$ that maximizes the mutual information $I(X;Y)$ with $P_Y(y)=\int P_{Y|X}(y|x)P_X(x)\text{dx}$ (this ...

**0**

votes

**0**answers

295 views

### Calculating entropy of adjacency matrix using eigenvalue decomposition?

How to calculate entropy using the eigenvalues when the eigenvalues are negative?
Is there a simple relation between the entropy of a matrix and its characteristic polynomial?

**3**

votes

**2**answers

158 views

### Estimate entropy of a binary process in terms of decay of correlations

Suppose $( X_{n} )$ is an ergodic binary process with
$$
\mathbb P(X_{n}=1)= \mathbb P(X_{n}=0)=\frac 12.
$$
Naturally the entropy (rate) $h(X)$ of $X=(X_{n})$ satisfies
$$
h(X)=\lim_{n\to\infty} ...

**9**

votes

**1**answer

1k views

### What's the maximum entropy probability distribution given bounds [a,b] and mean?

What is the continuous probability distribution that maximizes entropy, given only the bounds of the random variable [a,b] and the mean mu of the probability distribution?
For example:
if a=0, b=1, ...

**2**

votes

**0**answers

125 views

### A.G. Vitushkin's “Easily representable families of functions” - can it be generalized?

Background
In his monograph "Estimation of the complexity of the tabulation problem" (translated into English as "Theory of the Transmission and Processing of Information") Vitushkin studies ...

**2**

votes

**2**answers

253 views

### probability measures with entropy equal to nonnegative number

Is it true that for a given nonnegative number, there exists a measure-theoretical entropy value (supremum of entropies of all partitions under a measure-preserving transformation) that equals this ...

**4**

votes

**1**answer

363 views

### Upper bound on joint Renyi entropy

Renyi entropy of a random pair $(X,Y)$ with probability distribution $p_{X,Y}$ is defined by
\begin{equation}
H_\alpha(X,Y) = \frac{1}{1-\alpha}\log\sum_{x,y} p_{X,Y}(x,y)^\alpha.
\end{equation}
...

**4**

votes

**2**answers

280 views

### Maximum entropy priors in infinite dimensional spaces

Is there an extension of maximum entropy probability distributions for function spaces?
For $\mathbb{R}^n$ and discrete spaces, there is much literature about this problem under names such as ...

**0**

votes

**1**answer

221 views

### Entropy of inverse map for endomorphism case on surfaces

Hi,
I know that in the diffeomorphism case the measure entropy of the T:M^{2}-->M^{2} (M smooth Rimannian surface) will be the same as the measure entropy of T^{-1}. But i need to know about the ...

**3**

votes

**1**answer

677 views

### Convergence of Markov chains in terms of relative entropy

Consider a finite state, irreducible Markov chain with a rate matrix $Q$ and a stationary distribution $\pi$. Suppose the chain starts with the initial distribution $p$ at time $0$, then at time $t$ ...

**3**

votes

**1**answer

176 views

### Ising entropy of a finite L_1 x L_2 lattice

We know the entropy per site of the 2-d Ising model from Onsager's solution.
Has anybody also calculated the entropy for a finite rectangle of size L_1 x L_2
with periodic boundary conditions (i.e. on ...

**5**

votes

**1**answer

686 views

### Bounding the entropy of a convolution

Say we have a function $f:\mathbb{Z}_2^n \to \mathbb{R}$, such that $\sum _{x\in \mathbb{Z}_2^n} f(x)^2 = 1$ (so we can think of $\{ f(x)^2\} _{x\in \mathbb{Z}_2^n}$ as a distribution). It is natural ...

**4**

votes

**2**answers

652 views

### Is there a simple relation between the entropy of a matrix and its characteristic polynomial?

Assume $M$ is an invertible positive matrix of rank $N$. The Von Neumann entropy $H$ of a matrix $M$ with eigenvalues $\{ \lambda_n \}$ is
$H[M] = -\sum_{n=1}^N \lambda_n \ln \lambda_n$.
In ...

**1**

vote

**0**answers

142 views

### Entropy of factors of Bernoulli schemes

Let $X$ be a Bernoulli scheme. A factor $\psi: X \to Y$ is finitary if for almost every $x \in X$ there exist integers $m \leq n$ such that the zero coordinates of $\psi(x)$ and $\psi(x')$ agree for ...

**1**

vote

**0**answers

56 views

### Information amount of fuzzy data transfer [closed]

Suppose we have binary channel from which we are able to receive zeroes and ones. We also know apriory probability $p$ of receiving "1". Then we can calculate information amount of each digit $q$ we ...

**10**

votes

**2**answers

1k views

### Proving a messy inequality

EDIT:
After much work I was able to reduce the inequality to a single variable function which I need to show is non-positive. That function is (for $0\leq p\leq\frac{1}{2}$)
$$\frac{p^2(\log(p))^2 - ...

**2**

votes

**0**answers

299 views

### Does this inequality of negative relative entropy and quantum relative entropy hold?

Hello, everyone!
Question
I have a question about the relationship between general relative entropy and general quantum relative entropy: Given a unit vector $|i\rangle$ and two Hermitian matrices ...

**7**

votes

**2**answers

339 views

### Entropy conjecture for distributions over $\mathbb{Z}_n$

Suppose we have two independent random variables $X$ (with distribution $p_X$) and $Y$ (with distribution $p_Y$) which take values in the cyclic group $\mathbb{Z}_n$. Let $Z = X +Y$, where the ...

**1**

vote

**0**answers

247 views

### Incremental computation of a conditional entropy

Is it possible to compute a conditional entropy (see the two following formulas) in an incremental manner ? That is, the sets C and K are not fix: each time we have a new element c, set K may increase ...

**1**

vote

**1**answer

143 views

### asymptotic behaviour of the entropy and degeneracy

For each $n \in \mathbb{N}$ let $X_n$ be a random variable taking its values in a finite set $E_n$ with $P(X_n=x_n)>0$ for all $x_n \in E_n$. Say that $X_n$ is asymptotically degenerate if ...

**4**

votes

**1**answer

473 views

### Low degree polynomial approximation for the entropy function

Let $X$ be a discrete random variable with possible values
$\{x_1,\ldots,x_n\}$, and let $p$ denote the probability mass function of
$X$. In addition, denote $p_i=p(x_i)$.
The entropy of $X$ is ...

**1**

vote

**1**answer

377 views

### existence of finitely additive measures with zero entropy

Let $X$ be a countable set and $\mathcal M(X)$ be the set of finitely additive probability measures on $X$. If $\mu\in\mathcal M(X)$, I define the entropy of $\mu$ to be
$$
...

**2**

votes

**1**answer

234 views

### Compact group extension of a zero entropy system.

Suppose $T: X \to X$ is a continuous map
and $\mu$ a $T$-ergodic probability measure over the
Borel sets of $X$.
Now, suppose $K \subset \mathrm{Hom}(X)$ is a compact group
of measure-preserving ...

**10**

votes

**1**answer

1k views

### System with invariant measure, but no ergodic measure.

Question
Examples of continuous transformations $T: X \to X$ such that the family of invariant probability measures $M(T)$ is NOT empty but there is no ergodic measure ($E(T) = \emptyset$).
Notice ...

**10**

votes

**3**answers

769 views

### Supremum amongst Kolmogorov-Sinai entropies: ergodic or just invariant measures.

Cases where
$sup_{\mu \in E(T)} h_\mu(T)
\neq
\sup_{\mu \in M(T)} h_\mu(T)$.
Background
For a topological space $X$,
let $T: X \to X$ be a continuous application.
Then, call the set of ...

**0**

votes

**1**answer

374 views

### closed form expression for Rényi entropy for multivariate Gaussian distributions

Is there any closed form expression for Rényi entropy of a set variables with multivariate Gaussian distribution?

**0**

votes

**1**answer

308 views

### Robust entropy-like measure for analyzing uncertainity

I'm looking for a measure to analysis the uncertainty observed in a set of variables (with multivariate Gaussian distribution). So, I've tried conventional Shanon entropy (differential entropy) which ...

**3**

votes

**1**answer

5k views

### How to compute KL-divergence when PMF contains 0s?

From the Wikipedia page on Kullback-Leibler divergence, the way to compute this metric is to utilize the following formula:
The way I understand this is to compute the PMFs of two given sample sets ...

**6**

votes

**1**answer

241 views

### A geometric/topology notion of Typical Sequences? Power of typical sequences in multiuser channels?

The idea of Typical sequences(http://en.wikipedia.org/wiki/Typical_set) is a crucial concept in Shannon's proof of the Noisy channel coding theorem. Unfortunately the notion is not sufficient to ...

**8**

votes

**1**answer

1k views

### Entropy of the Ising model

Consider the standard Ising model on $[0,N]^2$ for $N$ large. By that I mean the square-lattice Ising model without external field, inside an $N$-by-$N$ square. What is its entropy for $N$ large? It ...

**4**

votes

**1**answer

389 views

### Infinitely Divisible Distributions and Maximal Entropy

The normal distribution on $\mathbb{R}$, the exponential distribution on $\mathbb{R}_{\geq 0}$, and the geometric distribution on $\mathbb{N}$ are examples of distributions that are both infinitely ...

**6**

votes

**1**answer

943 views

### Entropy of first return map and suspension flows

There are some well know formulas of Abramov about derived systems.
Firstly let $(X,\mu,f)$ be a probability preserving system and $A\subset X$ is measurable such that $\bigcup_{n\ge0}f^nA=X$. Let ...

**5**

votes

**2**answers

277 views

### Entropy of nested compact invariant sets

Let $f$ be a homeomorphism on a compact metric space $X$. $K_1\supset K_2\supset\cdots \supset K$ are compact subsets of $X$ such that $f(K_n)=K_n$ and
$K=\bigcap K_n$. If $h(f, K_1)<\infty$, do we ...

**1**

vote

**0**answers

86 views

### Median entropy to observe evolution of system?

Hello,
I am studying a dynamical system that takes as an initial condition a list. I want to analyze the evolution of Shannon's entropy in this system. I know the maximum entropy (50) and the minimum ...

**0**

votes

**2**answers

5k views

### How to ensure the non-negativity of Kullback-Leibler Divergence KLD Metric (Relative Entropy)?

Hello,
I’m having some problems in ensuring the non-negativity of KLD!
I know that KLD is always positive and I went over the proof. However, it doesn’t seem to work for me. In some cases I’m ...

**3**

votes

**2**answers

1k views

### Optimization of relative entropy

Wondering if my following question is an application of information theory:
Lets say we have a factory and ship boxes of stuff outside. If a competitor stands outside my factory, observes the stream ...

**3**

votes

**0**answers

624 views

### Entropy conjecture for flows

The entropy conjecture for diffeomorphisms (see for example this paper) asserts that for diffeomorphisms of manifolds, the log of the spectral radius of the actions of the diffeomorphism on the ...

**1**

vote

**3**answers

2k views

### Entropy of Random Signal

How would you proof that for all equal variance (Continuous) Random Signals, a Gaussian one would have the largest Entropy. Or in other words, Given a Variance Gaussian PDF maximizes the Entropy.
...