The tag has no usage guidance.

learn more… | top users | synonyms

1
vote
1answer
35 views

Maximizing joint entropy?

I'm stuck trying to find the maximum entropy probability distribution taking into account a joint distribution. Basically, I want to find the maximum entropy expression for $p(x,y)$ when the marginal ...
2
votes
1answer
542 views

Calculate channel capacity of general channel under constraint

Given a conditional distribution $P_{Y|X}$ I'd like to find the prior distribution $P_X$ that maximizes the mutual information $I(X;Y)$ with $P_Y(y)=\int P_{Y|X}(y|x)P_X(x)\text{dx}$ (this corresponds ...
4
votes
1answer
219 views

Do binary symmetric channels maximize mutual information?

Consider the following setup: $(X, Y)$ is a doubly symmetric binary source with parameter $0 < p < 1/2$, i.e., $X \sim \text{Bernoulli}(1/2)$, $Z \sim \text{Bernoulli}(p)$ and $Y = X \oplus Z$. ...
3
votes
3answers
215 views

Maximizing entropy under constraints

This question is about an extension of the variational principle in thermodynamical formalism when one adds linear constraints to the measures. Consider the one-sided shift ...
7
votes
2answers
303 views

Maximal entropy distribution with given conditionals

It is well known that of all the joint distributions $p(x,y)$ with fixed marginals $p(x),p(y)$, the one with the highest entropy is: $$ p(x,y)=p(x)p(y). $$ Suppose instead that we have conditionals. ...
-1
votes
1answer
84 views

Proving maximal entropy [closed]

It is quite easy to prove that $$H(S) \leq \log_2(|A|),$$ where $A$ is the number of events, using the Jensen inequality $$H(S) = E_S[\log_2(\frac{1}{P_S(s)})]\leq \log_2(E_S[(\frac{1}{P_S(s)})]) ...
2
votes
1answer
119 views

About Renyi entropy

If one is given a joint probability distribution over a finite set of discrete random variables then I guess there a notion of $\alpha-$Renyi entropy defined for it as $S_\alpha (X_1,..,X_n) = ...
3
votes
0answers
75 views

An inequality involving conditional variance and its connection to information theory

Given absolutely continuous random variables $(X, Y)$ with joint distribution $P_{XY}$, we construct $Z:=\sqrt{\gamma} Y+N_\mathsf{G}$ where $N_\mathsf{G}\sim N(0, 1)$ and is independent of $(X,Y)$ ...
7
votes
1answer
812 views

Bounding the entropy of a convolution

Say we have a function $f:\mathbb{Z}_2^n \to \mathbb{R}$, such that $\sum _{x\in \mathbb{Z}_2^n} f(x)^2 = 1$ (so we can think of $\{ f(x)^2\} _{x\in \mathbb{Z}_2^n}$ as a distribution). It is natural ...
2
votes
0answers
37 views

TAP expression for entropy [closed]

This paper by Barton and Cocco: http://www.phys.ens.fr/~cocco/Art/articlejstat.pdf claims on page 17 (Formula (30)) an expression for the "high-temperature" entropy of an Ising model, given its ...
1
vote
0answers
95 views

weak-* versus entropy growth

General question. Let $\eta_{n}$ be a sequence of invariant measures on $\{0,1,2,...,p-1\}^{\mathbb{N}}$ and $B$ the Bernoulli uniform measure. Knowing that $\eta_{n} \rightarrow B$ in the weak-* ...
30
votes
2answers
2k views

An Entropy Inequality

Let $X,Y$ be probability measures on $\{1,2,\dots,n\}$, and set $K=\sum_i\sqrt{X(i)Y(i)}$ so that $Z:=\frac{1}{K}\sqrt{XY}$ is also a probability measure on $\{1,2,\dots,n\}$. How can we prove the ...
6
votes
1answer
232 views

Finding discrete entropy via differential entropy

In a recent math.se question the following was asked which I have slightly edited. " Consider a fixed and given $n$ by $n$ matrix $M$ whose elements are chosen from $\{-1,1\}$. Consider also a random ...
1
vote
0answers
93 views

Strict factor of a dynamical system with the same entropy [closed]

Say that a factor of an invertible measure-preserving transformation $T$ is strict if it is not isomorphic to $T$. Does there exist an invertibe mpt $T$ such that $0 < h(T) < \infty$ and ...
7
votes
2answers
215 views

Estimating entropy conditional to an event

Take for example the measure $\mu(n)=n^2$ on $\{1, \ldots, N\}$ and a random variable $X$ distributed according to the probability obtained by normalizing $\mu$. Does there exists a constant ...
0
votes
2answers
7k views

How to ensure the non-negativity of Kullback-Leibler Divergence KLD Metric (Relative Entropy)?

Hello, I’m having some problems in ensuring the non-negativity of KLD! I know that KLD is always positive and I went over the proof. However, it doesn’t seem to work for me. In some cases I’m ...
3
votes
1answer
256 views

Convolution of measures - entropy growth

Imagine you have two shift-invariant measures $\mu, \nu$ in the Bernoulli space $\{0,1\}^{\mathbb{N}}$ with positive entropy and both are not the Bernoulli measure $(\frac{1}{2},\frac{1}{2})$. I know ...
5
votes
0answers
62 views

Basis functions for approximation of a convex function on unit simplex

Consider the unit $D$-simplex $S^D=\left\lbrace (x_0, x_1, \ldots, x_D) \in \mathbb{R}^{D+1} \mid \sum\limits_{i=0}^{D}x_i = 1, x_i \geq 0 \right\rbrace$. I have a bounded, convex function ...
0
votes
1answer
112 views

entropy growth of invariant measures - General question

In general, given a sequence of shift-invariant measures $\eta_{n}$ on $\{0,1\}^{\mathbb{N}}$ what to do to guarantee this convergence of entropies: $$h(\eta_{n}) \rightarrow \log2?$$ Because I'm ...
1
vote
1answer
116 views

entropy and d-bar: how do we estimate continuity?

Let $G = \{0,1\}^{\mathbb{N}} = \mathbb{Z}_{2}^{\mathbb{N}}$ be the Bernoulli space of two symbols, let $\sigma$ be the shift map and $M(G)$ the set of $\sigma$-invariant probabilities. Let $\bar{d}$ ...
4
votes
2answers
256 views

Entropy equals zero?

Imagine you have a shift invariant ($\sigma$-invariant) probability measure $\eta$ in the Bernoulli space $\{0,1\}^{\mathbb{N}}$. Define $\mathcal{P} = \{[0],[1]\}$; $\mathcal{P}^{n} = ...
4
votes
1answer
422 views

Infinitely Divisible Distributions and Maximal Entropy

The normal distribution on $\mathbb{R}$, the exponential distribution on $\mathbb{R}_{\geq 0}$, and the geometric distribution on $\mathbb{N}$ are examples of distributions that are both infinitely ...
0
votes
1answer
140 views

joining or coupling

given two shift invariant measures in the Bernoulli space $\{0,1\}^{\mathbb{N}}$, is there a way to construct joinings of them? It's very diffcult, in general, to find exactly the minimal joining i.e, ...
5
votes
2answers
225 views

General additive function of probability

Let $H$ be a function of finite sequences of probabilities (non-negative numbers summing up to 1) into real numbers, such that: $H$ is continuous, $H$ is symmetric w.r.t. the order of its arguments, ...
0
votes
0answers
89 views

Entropy, convergence and invariant measures

Could you give conditions that a sequence of shift invariant measures $\eta_{n}$ has to satisfy in order to happen this convergence in terms of entropies $h(\eta_{n})$: $h(\eta_{n}) \rightarrow ...
11
votes
1answer
1k views

System with invariant measure, but no ergodic measure.

Question Examples of continuous transformations $T: X \to X$ such that the family of invariant probability measures $M(T)$ is NOT empty but there is no ergodic measure ($E(T) = \emptyset$). Notice ...
1
vote
1answer
172 views

Entropy, Convergence

imagine you have a sequence $\eta_{n}$ of (shift) invariant measures in the Bernoulli space $\{0,1\}^{\mathbb{N}}$ that satisfy the following: there are a $0<\delta <1$ and an $N$ such that $$n ...
3
votes
1answer
262 views

Information theory from negative probability

Szekely provides a convincing argument of negative probability here: http://www.wilmott.com/pdfs/100609_gjs.pdf What does a reformulation of classical information theory built from negative ...
2
votes
2answers
273 views

Entropy inequality

Let $P,Q$ be probabilities on a finite set $A$ with $Q(a)\gt 0$, for all $a\in A$, and let $H(P)$, $H(Q)$ denote their entropies and $D(P\,\|\,Q)$ denote their Kullback-Leibler distance. Is it always ...
0
votes
1answer
81 views

Asymptotically full stationary process

Let $(X_n)_{n \in \mathbb{Z}}$ be a stationary process on a finite set $A$. Say that it is asymptotically full if for every increasing sequence of subsets $B_n \subset A^n$ such that ...
2
votes
2answers
587 views

Convexity and semicontinuity of the relative entropy function

There are several different definitions of relative entropy, and some of them are not equivalent. Following is the definition we will use in this question. Let $M$ be a closed manifold and ...
1
vote
0answers
114 views

Estimates of entropy of functional spaces

Let $M^n$ be a compact $n$-dimensional manifold. For $k\geq 0$ let us denote by $C^k(M)$ the Banach space of $k$ times continuously differentiable functions, and $B_{C^k}$ denote the unit ball of it. ...
2
votes
1answer
198 views

Embeddings of subshifts

Consider $(X,\sigma_X)$ and $(Y, \sigma_{Y})$ be subshifts of the one sided shift in two symbols. Assume that $(X,\sigma_X)$ is a transitive subshift of finite type and $(Y, \sigma_{Y})$ is a ...
9
votes
2answers
474 views

Expected centered entropy of the binomial distribution

In short, the function I am interested in is the following: $$I_n(p) = \sum_{k=0}^n \binom{n}{k} p^k (1-p)^{n-k} \left[h(p) - h\left(\tfrac{k}{n}\right)\right],$$ where $h(x) \triangleq -x \log x - ...
1
vote
1answer
357 views

Coupon Collector Problem for Non-Uniform Coupons: Bound on the number of missed Coupons

Suppose $\mathcal B=\{1,2,..,b\}$ is the set of all possible coupons, with $\mathbf p = ( p_1,p_2,...,p_b)$ assigning the probability of occurrence for all coupons in $\mathcal B$. The "traditional ...
2
votes
0answers
96 views

Concentration bound in high min entropy distribution

Let $(X_{1},\dots,X_{m})$ be joint distribution on $\{0,1\}^{m}$ with that $H_{\infty}(X_{1},\cdots,X_{m})\geq m-r$, where $H_{\infty}$ means min-entropy. Let $P_{1},...,P_{n}\subseteq [m]$ be sets ...
17
votes
8answers
2k views

When do people actually use the maximum entropy distribution?

One of the standard problems in convex optimization is the calculation of the maximum entropy distribution that satisfies some set of criteria. For example, if $\mathbf{x} \in \mathbb R^n$ is an ...
16
votes
3answers
584 views

Intrinsic significance of differential entropy

Many commentators (e.g. Jaynes, Rota) argue that the notion of "differential entropy" is problematic (as commonly defined by $ h(X) = \int ( \log\frac{1}{p(x)} ) p(x) \, dx $, where $X$ is a random ...
1
vote
1answer
420 views

Discrete Maximum Entropy Distribution with given mean

For a given mean $\mu$, what is the entropy maximizing probability distribution on the nonnegative integers? Different sources indicated either the geometric or the Poisson distribution for this. As ...
8
votes
2answers
455 views

Inequality in information theory

I am reading the paper "chain independence and common information" (http://ttic.uchicago.edu/~yury/papers/independ.pdf). In this paper, an inequality is used several times (without proof) which looks ...
2
votes
0answers
240 views

Interpretation of Shannon Entropy Application

Consider a collection of increasing positive integers $\{a_i\}_{i=1}^m$ and the distribution $p_i=\frac{a_i}{\sum_{i=1}^ma_i}$. Let entropy of $\mathcal{A}=\{a_i\}_{i=1}^m$ be given by ...
2
votes
0answers
70 views

Sequence transformations that are entropy invariant

Consider a collection of increasing positive integers $\{a_i\}_{i=1}^m$ and the distribution $p_i=\frac{a_i}{\sum_{i=1}^ma_i}$. Define entropy of $\mathcal{A}=\{a_i\}_{i=1}^m$ by ...
1
vote
1answer
111 views

Limiting Entropy of deterministic sequences - 1

Consider a collection of increasing positive integers $\{a_i\}_{i=1}^m$. Consider distribution $p_i=\frac{a_i}{\sum_{i=1}^ma_i}$. Given $\{a_i\}_{i=1}^m$, let $\mathcal{P}_{a,m}$ be distribution at ...
1
vote
0answers
44 views

Limiting Entropy of deterministic sequences - 2

Consider a collection of increasing positive integers $\{a_i\}_{i=1}^m$ and the distribution $p_i=\frac{a_i}{\sum_{i=1}^ma_i}$. Given $\{a_i\}_{i=1}^m$, let $\mathcal{P}_a$ be limiting distribution ...
1
vote
1answer
86 views

Entropy difference dominance of sequences

Consider a collection of positive integers $\{a_i\}_{i=1}^m$ and the distribution $p_i=\frac{a_i}{\sum_{i=1}^ma_i}$. Similarly for the collection $\{a_i\}_{i=1}^{m+1}$ form the distribution ...
1
vote
1answer
59 views

Entropy dominance of certain restricted sequenes

Say you have positive $\{a_i\}_{i=1}^n$ and you have $p_i=\frac{a_i}{\sum_{i=1}^na_i}$, then assume you have a $C$ such that $C<2a_n\ll\sum_{i=1}^na_i$ (that is $C$ is not very large), then define ...
2
votes
1answer
81 views

Entropy dominance

Let $0<a<b<c$ be distinct positive reals. Define four different probability distributions: $$\mathcal{P}_{ab}:P_{a,ab}=\frac{a}{a+b}=1-P_{b,ab}$$ ...
8
votes
1answer
232 views

Higher moments of information and Renyi entropy

For a given discrete probability distribution, Shannon entropy can be though as an expectation value $\langle - \log p \rangle$ (see also: What is entropy, really?, What is the role of the logarithm ...
2
votes
0answers
105 views

Property of relative entropy [closed]

For $X$ a measurable space and $P,Q$ two probability measure on $X$ s.t. $Q$ is absolutely continuous with respect to $P$, the relative entropy is defined as $$D(Q\|P)=\int_X ...
3
votes
1answer
134 views

Average entropy of quantum system in bipartite pure state for finite temperature

[I got halfway through writing this when I found the paper that answers the question in (essentially) the affirmative. I'll post it anyways in case anyone is interested.] Background: If a random ...