The tag has no usage guidance.

learn more… | top users | synonyms

1
vote
1answer
31 views

Maximizing joint entropy?

I'm stuck trying to find the maximum entropy probability distribution taking into account a joint distribution. Basically, I want to find the maximum entropy expression for $p(x,y)$ when the marginal ...
-1
votes
1answer
84 views

Proving maximal entropy [closed]

It is quite easy to prove that $$H(S) \leq \log_2(|A|),$$ where $A$ is the number of events, using the Jensen inequality $$H(S) = E_S[\log_2(\frac{1}{P_S(s)})]\leq \log_2(E_S[(\frac{1}{P_S(s)})]) ...
2
votes
1answer
119 views

About Renyi entropy

If one is given a joint probability distribution over a finite set of discrete random variables then I guess there a notion of $\alpha-$Renyi entropy defined for it as $S_\alpha (X_1,..,X_n) = ...
3
votes
0answers
75 views

An inequality involving conditional variance and its connection to information theory

Given absolutely continuous random variables $(X, Y)$ with joint distribution $P_{XY}$, we construct $Z:=\sqrt{\gamma} Y+N_\mathsf{G}$ where $N_\mathsf{G}\sim N(0, 1)$ and is independent of $(X,Y)$ ...
2
votes
0answers
37 views

TAP expression for entropy [closed]

This paper by Barton and Cocco: http://www.phys.ens.fr/~cocco/Art/articlejstat.pdf claims on page 17 (Formula (30)) an expression for the "high-temperature" entropy of an Ising model, given its ...
1
vote
0answers
95 views

weak-* versus entropy growth

General question. Let $\eta_{n}$ be a sequence of invariant measures on $\{0,1,2,...,p-1\}^{\mathbb{N}}$ and $B$ the Bernoulli uniform measure. Knowing that $\eta_{n} \rightarrow B$ in the weak-* ...
1
vote
0answers
93 views

Strict factor of a dynamical system with the same entropy [closed]

Say that a factor of an invertible measure-preserving transformation $T$ is strict if it is not isomorphic to $T$. Does there exist an invertibe mpt $T$ such that $0 < h(T) < \infty$ and ...
6
votes
1answer
232 views

Finding discrete entropy via differential entropy

In a recent math.se question the following was asked which I have slightly edited. " Consider a fixed and given $n$ by $n$ matrix $M$ whose elements are chosen from $\{-1,1\}$. Consider also a random ...
0
votes
1answer
112 views

entropy growth of invariant measures - General question

In general, given a sequence of shift-invariant measures $\eta_{n}$ on $\{0,1\}^{\mathbb{N}}$ what to do to guarantee this convergence of entropies: $$h(\eta_{n}) \rightarrow \log2?$$ Because I'm ...
1
vote
1answer
116 views

entropy and d-bar: how do we estimate continuity?

Let $G = \{0,1\}^{\mathbb{N}} = \mathbb{Z}_{2}^{\mathbb{N}}$ be the Bernoulli space of two symbols, let $\sigma$ be the shift map and $M(G)$ the set of $\sigma$-invariant probabilities. Let $\bar{d}$ ...
5
votes
0answers
62 views

Basis functions for approximation of a convex function on unit simplex

Consider the unit $D$-simplex $S^D=\left\lbrace (x_0, x_1, \ldots, x_D) \in \mathbb{R}^{D+1} \mid \sum\limits_{i=0}^{D}x_i = 1, x_i \geq 0 \right\rbrace$. I have a bounded, convex function ...
4
votes
2answers
256 views

Entropy equals zero?

Imagine you have a shift invariant ($\sigma$-invariant) probability measure $\eta$ in the Bernoulli space $\{0,1\}^{\mathbb{N}}$. Define $\mathcal{P} = \{[0],[1]\}$; $\mathcal{P}^{n} = ...
7
votes
2answers
214 views

Estimating entropy conditional to an event

Take for example the measure $\mu(n)=n^2$ on $\{1, \ldots, N\}$ and a random variable $X$ distributed according to the probability obtained by normalizing $\mu$. Does there exists a constant ...
0
votes
1answer
140 views

joining or coupling

given two shift invariant measures in the Bernoulli space $\{0,1\}^{\mathbb{N}}$, is there a way to construct joinings of them? It's very diffcult, in general, to find exactly the minimal joining i.e, ...
4
votes
1answer
219 views

Do binary symmetric channels maximize mutual information?

Consider the following setup: $(X, Y)$ is a doubly symmetric binary source with parameter $0 < p < 1/2$, i.e., $X \sim \text{Bernoulli}(1/2)$, $Z \sim \text{Bernoulli}(p)$ and $Y = X \oplus Z$. ...
0
votes
0answers
89 views

Entropy, convergence and invariant measures

Could you give conditions that a sequence of shift invariant measures $\eta_{n}$ has to satisfy in order to happen this convergence in terms of entropies $h(\eta_{n})$: $h(\eta_{n}) \rightarrow ...
1
vote
1answer
172 views

Entropy, Convergence

imagine you have a sequence $\eta_{n}$ of (shift) invariant measures in the Bernoulli space $\{0,1\}^{\mathbb{N}}$ that satisfy the following: there are a $0<\delta <1$ and an $N$ such that $$n ...
3
votes
1answer
261 views

Information theory from negative probability

Szekely provides a convincing argument of negative probability here: http://www.wilmott.com/pdfs/100609_gjs.pdf What does a reformulation of classical information theory built from negative ...
7
votes
2answers
302 views

Maximal entropy distribution with given conditionals

It is well known that of all the joint distributions $p(x,y)$ with fixed marginals $p(x),p(y)$, the one with the highest entropy is: $$ p(x,y)=p(x)p(y). $$ Suppose instead that we have conditionals. ...
0
votes
1answer
81 views

Asymptotically full stationary process

Let $(X_n)_{n \in \mathbb{Z}}$ be a stationary process on a finite set $A$. Say that it is asymptotically full if for every increasing sequence of subsets $B_n \subset A^n$ such that ...
3
votes
1answer
256 views

Convolution of measures - entropy growth

Imagine you have two shift-invariant measures $\mu, \nu$ in the Bernoulli space $\{0,1\}^{\mathbb{N}}$ with positive entropy and both are not the Bernoulli measure $(\frac{1}{2},\frac{1}{2})$. I know ...
3
votes
3answers
215 views

Maximizing entropy under constraints

This question is about an extension of the variational principle in thermodynamical formalism when one adds linear constraints to the measures. Consider the one-sided shift ...
2
votes
2answers
273 views

Entropy inequality

Let $P,Q$ be probabilities on a finite set $A$ with $Q(a)\gt 0$, for all $a\in A$, and let $H(P)$, $H(Q)$ denote their entropies and $D(P\,\|\,Q)$ denote their Kullback-Leibler distance. Is it always ...
1
vote
0answers
114 views

Estimates of entropy of functional spaces

Let $M^n$ be a compact $n$-dimensional manifold. For $k\geq 0$ let us denote by $C^k(M)$ the Banach space of $k$ times continuously differentiable functions, and $B_{C^k}$ denote the unit ball of it. ...
2
votes
1answer
197 views

Embeddings of subshifts

Consider $(X,\sigma_X)$ and $(Y, \sigma_{Y})$ be subshifts of the one sided shift in two symbols. Assume that $(X,\sigma_X)$ is a transitive subshift of finite type and $(Y, \sigma_{Y})$ is a ...
9
votes
2answers
473 views

Expected centered entropy of the binomial distribution

In short, the function I am interested in is the following: $$I_n(p) = \sum_{k=0}^n \binom{n}{k} p^k (1-p)^{n-k} \left[h(p) - h\left(\tfrac{k}{n}\right)\right],$$ where $h(x) \triangleq -x \log x - ...
1
vote
1answer
355 views

Coupon Collector Problem for Non-Uniform Coupons: Bound on the number of missed Coupons

Suppose $\mathcal B=\{1,2,..,b\}$ is the set of all possible coupons, with $\mathbf p = ( p_1,p_2,...,p_b)$ assigning the probability of occurrence for all coupons in $\mathcal B$. The "traditional ...
2
votes
0answers
96 views

Concentration bound in high min entropy distribution

Let $(X_{1},\dots,X_{m})$ be joint distribution on $\{0,1\}^{m}$ with that $H_{\infty}(X_{1},\cdots,X_{m})\geq m-r$, where $H_{\infty}$ means min-entropy. Let $P_{1},...,P_{n}\subseteq [m]$ be sets ...
1
vote
1answer
419 views

Discrete Maximum Entropy Distribution with given mean

For a given mean $\mu$, what is the entropy maximizing probability distribution on the nonnegative integers? Different sources indicated either the geometric or the Poisson distribution for this. As ...
2
votes
0answers
70 views

Sequence transformations that are entropy invariant

Consider a collection of increasing positive integers $\{a_i\}_{i=1}^m$ and the distribution $p_i=\frac{a_i}{\sum_{i=1}^ma_i}$. Define entropy of $\mathcal{A}=\{a_i\}_{i=1}^m$ by ...
2
votes
0answers
239 views

Interpretation of Shannon Entropy Application

Consider a collection of increasing positive integers $\{a_i\}_{i=1}^m$ and the distribution $p_i=\frac{a_i}{\sum_{i=1}^ma_i}$. Let entropy of $\mathcal{A}=\{a_i\}_{i=1}^m$ be given by ...
1
vote
0answers
44 views

Limiting Entropy of deterministic sequences - 2

Consider a collection of increasing positive integers $\{a_i\}_{i=1}^m$ and the distribution $p_i=\frac{a_i}{\sum_{i=1}^ma_i}$. Given $\{a_i\}_{i=1}^m$, let $\mathcal{P}_a$ be limiting distribution ...
1
vote
1answer
111 views

Limiting Entropy of deterministic sequences - 1

Consider a collection of increasing positive integers $\{a_i\}_{i=1}^m$. Consider distribution $p_i=\frac{a_i}{\sum_{i=1}^ma_i}$. Given $\{a_i\}_{i=1}^m$, let $\mathcal{P}_{a,m}$ be distribution at ...
1
vote
1answer
86 views

Entropy difference dominance of sequences

Consider a collection of positive integers $\{a_i\}_{i=1}^m$ and the distribution $p_i=\frac{a_i}{\sum_{i=1}^ma_i}$. Similarly for the collection $\{a_i\}_{i=1}^{m+1}$ form the distribution ...
1
vote
1answer
59 views

Entropy dominance of certain restricted sequenes

Say you have positive $\{a_i\}_{i=1}^n$ and you have $p_i=\frac{a_i}{\sum_{i=1}^na_i}$, then assume you have a $C$ such that $C<2a_n\ll\sum_{i=1}^na_i$ (that is $C$ is not very large), then define ...
2
votes
1answer
81 views

Entropy dominance

Let $0<a<b<c$ be distinct positive reals. Define four different probability distributions: $$\mathcal{P}_{ab}:P_{a,ab}=\frac{a}{a+b}=1-P_{b,ab}$$ ...
8
votes
1answer
232 views

Higher moments of information and Renyi entropy

For a given discrete probability distribution, Shannon entropy can be though as an expectation value $\langle - \log p \rangle$ (see also: What is entropy, really?, What is the role of the logarithm ...
2
votes
0answers
105 views

Property of relative entropy [closed]

For $X$ a measurable space and $P,Q$ two probability measure on $X$ s.t. $Q$ is absolutely continuous with respect to $P$, the relative entropy is defined as $$D(Q\|P)=\int_X ...
17
votes
8answers
2k views

When do people actually use the maximum entropy distribution?

One of the standard problems in convex optimization is the calculation of the maximum entropy distribution that satisfies some set of criteria. For example, if $\mathbf{x} \in \mathbb R^n$ is an ...
3
votes
1answer
134 views

Average entropy of quantum system in bipartite pure state for finite temperature

[I got halfway through writing this when I found the paper that answers the question in (essentially) the affirmative. I'll post it anyways in case anyone is interested.] Background: If a random ...
2
votes
1answer
152 views

Which is the right way to compute the Approximate Entropy (ApEn)?

My problem is the inconsistency between the definition and the computation of the Approximate entropy (ApEn). Suppose $u = (u_i:1\leq i \leq N)$ is a sequence of ...
1
vote
1answer
116 views

Entropy on a draw from a random distribution.

Suppose I am attempting to calculate the entropy of a continuous, normally distributed random variable $X$, from the distribution $\mathcal{N}(\mu, \sigma)$. This is easy to to do - I just calculate ...
4
votes
0answers
101 views

$q$-deformations of fundamental equation of information and entropies

Classical information theory: fundamental equation of information In classical information theory, the information $I(A)$ of an event $A$ (any element of the $\sigma$-algebra $\mathcal F$ of a ...
1
vote
0answers
53 views

Characterization of the optimal solution in relative entropy minimization

The following optimization problem is related to relative entropy and to the limit of the iterative proportional fitting procedure. For $1 \leq i,j \leq n$ and fixed $w_{ij} \geq 0$, and fixed $a_i, ...
1
vote
1answer
181 views

Invariance of mutual information

Let $I(X,Y):=H(X)+H(Y)-H(X,Y)$ be the mutual information of the joint probability distribution $p_{XY}$ (here $H(\cdot)$ is the Shannon entropy of its argument). I know that the mutual information is ...
3
votes
0answers
128 views

On the volume entropy of negatively curved manifolds

Let $X$ be the universal cover of a closed negatively curved Riemannian manifold. Let $x_0\in X$ be a base point, $S$ be the unit sphere in $T_{x_0}X$ and $\exp:T_{x_0}X\rightarrow X$ be the ...
0
votes
1answer
130 views

Why private randomization does not help the Shannon's source coding

I am wondering why stochastic encoder and decoder can not help the Shannon source coding? I know the achievability scheme of source coding, which is based on typicality, is deterministic, and hence we ...
9
votes
2answers
427 views

Entropy for Haar measure on $O(n)$

Let $G$ be a locally compact group. A measure $\mu$ is the right-Haar measure on $G$ if for every $g\in G$ and $E\subseteq G$ Borel set $\mu(Eg)=\mu(E)$. It is known that every locally compact group ...
16
votes
3answers
584 views

Intrinsic significance of differential entropy

Many commentators (e.g. Jaynes, Rota) argue that the notion of "differential entropy" is problematic (as commonly defined by $ h(X) = \int ( \log\frac{1}{p(x)} ) p(x) \, dx $, where $X$ is a random ...
1
vote
3answers
160 views

How to estimate the entropy of a distribution on a power set?

Given a probability distribution $(X,p)$, its entropy is defined as $H=-\sum_{x\in X} p(x)\log p(x)$. Given a sample of observations $x_n,n=1..N$, one can estimate $p(x)=\frac{\#\{i:x_i=x\}}{N}$ and ...