The tag has no wiki summary.

learn more… | top users | synonyms

3
votes
1answer
588 views

Convergence of Markov chains in terms of relative entropy

Consider a finite state, irreducible Markov chain with a rate matrix $Q$ and a stationary distribution $\pi$. Suppose the chain starts with the initial distribution $p$ at time $0$, then at time $t$ ...
3
votes
1answer
170 views

Ising entropy of a finite L_1 x L_2 lattice

We know the entropy per site of the 2-d Ising model from Onsager's solution. Has anybody also calculated the entropy for a finite rectangle of size L_1 x L_2 with periodic boundary conditions (i.e. on ...
5
votes
1answer
640 views

Bounding the entropy of a convolution

Say we have a function $f:\mathbb{Z}_2^n \to \mathbb{R}$, such that $\sum _{x\in \mathbb{Z}_2^n} f(x)^2 = 1$ (so we can think of $\{ f(x)^2\} _{x\in \mathbb{Z}_2^n}$ as a distribution). It is natural ...
4
votes
2answers
577 views

Is there a simple relation between the entropy of a matrix and its characteristic polynomial?

Assume $M$ is an invertible positive matrix of rank $N$. The Von Neumann entropy $H$ of a matrix $M$ with eigenvalues $\{ \lambda_n \}$ is $H[M] = -\sum_{n=1}^N \lambda_n \ln \lambda_n$. In ...
1
vote
0answers
137 views

Entropy of factors of Bernoulli schemes

Let $X$ be a Bernoulli scheme. A factor $\psi: X \to Y$ is finitary if for almost every $x \in X$ there exist integers $m \leq n$ such that the zero coordinates of $\psi(x)$ and $\psi(x')$ agree for ...
1
vote
0answers
55 views

Information amount of fuzzy data transfer [closed]

Suppose we have binary channel from which we are able to receive zeroes and ones. We also know apriory probability $p$ of receiving "1". Then we can calculate information amount of each digit $q$ we ...
10
votes
2answers
1k views

Proving a messy inequality

EDIT: After much work I was able to reduce the inequality to a single variable function which I need to show is non-positive. That function is (for $0\leq p\leq\frac{1}{2}$) $$\frac{p^2(\log(p))^2 - ...
2
votes
0answers
286 views

Does this inequality of negative relative entropy and quantum relative entropy hold?

Hello, everyone! Question I have a question about the relationship between general relative entropy and general quantum relative entropy: Given a unit vector $|i\rangle$ and two Hermitian matrices ...
7
votes
2answers
326 views

Entropy conjecture for distributions over $\mathbb{Z}_n$

Suppose we have two independent random variables $X$ (with distribution $p_X$) and $Y$ (with distribution $p_Y$) which take values in the cyclic group $\mathbb{Z}_n$. Let $Z = X +Y$, where the ...
1
vote
0answers
244 views

Incremental computation of a conditional entropy

Is it possible to compute a conditional entropy (see the two following formulas) in an incremental manner ? That is, the sets C and K are not fix: each time we have a new element c, set K may increase ...
1
vote
1answer
141 views

asymptotic behaviour of the entropy and degeneracy

For each $n \in \mathbb{N}$ let $X_n$ be a random variable taking its values in a finite set $E_n$ with $P(X_n=x_n)>0$ for all $x_n \in E_n$. Say that $X_n$ is asymptotically degenerate if ...
4
votes
1answer
429 views

Low degree polynomial approximation for the entropy function

Let $X$ be a discrete random variable with possible values $\{x_1,\ldots,x_n\}$, and let $p$ denote the probability mass function of $X$. In addition, denote $p_i=p(x_i)$. The entropy of $X$ is ...
1
vote
1answer
372 views

existence of finitely additive measures with zero entropy

Let $X$ be a countable set and $\mathcal M(X)$ be the set of finitely additive probability measures on $X$. If $\mu\in\mathcal M(X)$, I define the entropy of $\mu$ to be $$ ...
2
votes
1answer
226 views

Compact group extension of a zero entropy system.

Suppose $T: X \to X$ is a continuous map and $\mu$ a $T$-ergodic probability measure over the Borel sets of $X$. Now, suppose $K \subset \mathrm{Hom}(X)$ is a compact group of measure-preserving ...
10
votes
1answer
1k views

System with invariant measure, but no ergodic measure.

Question Examples of continuous transformations $T: X \to X$ such that the family of invariant probability measures $M(T)$ is NOT empty but there is no ergodic measure ($E(T) = \emptyset$). Notice ...
10
votes
3answers
740 views

Supremum amongst Kolmogorov-Sinai entropies: ergodic or just invariant measures.

Cases where $sup_{\mu \in E(T)} h_\mu(T) \neq \sup_{\mu \in M(T)} h_\mu(T)$. Background For a topological space $X$, let $T: X \to X$ be a continuous application. Then, call the set of ...
0
votes
1answer
348 views

closed form expression for Rényi entropy for multivariate Gaussian distributions

Is there any closed form expression for Rényi entropy of a set variables with multivariate Gaussian distribution?
0
votes
1answer
297 views

Robust entropy-like measure for analyzing uncertainity

I'm looking for a measure to analysis the uncertainty observed in a set of variables (with multivariate Gaussian distribution). So, I've tried conventional Shanon entropy (differential entropy) which ...
2
votes
1answer
4k views

How to compute KL-divergence when PMF contains 0s?

From the Wikipedia page on Kullback-Leibler divergence, the way to compute this metric is to utilize the following formula: The way I understand this is to compute the PMFs of two given sample sets ...
6
votes
1answer
238 views

A geometric/topology notion of Typical Sequences? Power of typical sequences in multiuser channels?

The idea of Typical sequences(http://en.wikipedia.org/wiki/Typical_set) is a crucial concept in Shannon's proof of the Noisy channel coding theorem. Unfortunately the notion is not sufficient to ...
8
votes
1answer
1k views

Entropy of the Ising model

Consider the standard Ising model on $[0,N]^2$ for $N$ large. By that I mean the square-lattice Ising model without external field, inside an $N$-by-$N$ square. What is its entropy for $N$ large? It ...
3
votes
1answer
377 views

Infinitely Divisible Distributions and Maximal Entropy

The normal distribution on $\mathbb{R}$, the exponential distribution on $\mathbb{R}_{\geq 0}$, and the geometric distribution on $\mathbb{N}$ are examples of distributions that are both infinitely ...
6
votes
1answer
856 views

Entropy of first return map and suspension flows

There are some well know formulas of Abramov about derived systems. Firstly let $(X,\mu,f)$ be a probability preserving system and $A\subset X$ is measurable such that $\bigcup_{n\ge0}f^nA=X$. Let ...
5
votes
2answers
276 views

Entropy of nested compact invariant sets

Let $f$ be a homeomorphism on a compact metric space $X$. $K_1\supset K_2\supset\cdots \supset K$ are compact subsets of $X$ such that $f(K_n)=K_n$ and $K=\bigcap K_n$. If $h(f, K_1)<\infty$, do we ...
1
vote
0answers
85 views

Median entropy to observe evolution of system?

Hello, I am studying a dynamical system that takes as an initial condition a list. I want to analyze the evolution of Shannon's entropy in this system. I know the maximum entropy (50) and the minimum ...
0
votes
2answers
4k views

How to ensure the non-negativity of Kullback-Leibler Divergence KLD Metric (Relative Entropy)?

Hello, I’m having some problems in ensuring the non-negativity of KLD! I know that KLD is always positive and I went over the proof. However, it doesn’t seem to work for me. In some cases I’m ...
3
votes
2answers
932 views

Optimization of relative entropy

Wondering if my following question is an application of information theory: Lets say we have a factory and ship boxes of stuff outside. If a competitor stands outside my factory, observes the stream ...
3
votes
0answers
606 views

Entropy conjecture for flows

The entropy conjecture for diffeomorphisms (see for example this paper) asserts that for diffeomorphisms of manifolds, the log of the spectral radius of the actions of the diffeomorphism on the ...
1
vote
3answers
2k views

Entropy of Random Signal

How would you proof that for all equal variance (Continuous) Random Signals, a Gaussian one would have the largest Entropy. Or in other words, Given a Variance Gaussian PDF maximizes the Entropy. ...