# Questions tagged [entropy]

The entropy tag has no usage guidance.

266
questions

1
vote

0
answers

26
views

### A distribution $\pi \propto \exp(-f)$ satisfies log-Sobolev inequality, does $\exp(-af)$ also satisfy LSI?

Assume a distribution $\pi \propto e^{-f}$ satisfies log-Sobolev inequality (LSI)
$$\forall \rho \in P(\mathbb{R}^n), \quad KL(\rho\| \pi) \le \frac{1}{2\lambda} I(\rho \| \pi)$$
with LSI constant $\...

7
votes

2
answers

182
views

### Does entropy of the random walk control the return probability

Given an infinite connected graph $G$ of bounded degree with vertex set $X$, let $P_x^n$ the time $n$ distribution of the simple random walk started at the vertex $x$ (so $P^n_x(y)$ is the probability ...

4
votes

1
answer

230
views

### Bipartite version of Hamming bound (two families of codewords with large Hamming distance)?

Update: In light of Fedor Petrov's answer, I added an additional requirement that all strings in $A$ and $B$ have Hamming weight exactly $n/2$, which hopefully makes the question more interesting.
...

1
vote

1
answer

109
views

### Fokker-Planck: equivalence between linear spectral gap and nonlinear displacement convexity?

In a smooth, bounded and convex domain $\Omega\subset \mathbb R^d$ consider the usual linear Fokker-Planck equation with Neumann (some would say Robin) boundary conditions
\begin{equation}
\label{FP}
\...

0
votes

0
answers

30
views

### Relation between multivariate estimation error and differential entropy

On page 255 of the book "Elements of information theory" by Thomas M. Cover and Joy A. Thomas, there is a theorem: For any random variable $X$ and estimator $\hat{X}$, $$E(X-\hat{X})^2 \geq \...

1
vote

0
answers

57
views

### Polynomial entropy of topological dynamical systems

For a continuous mapping $\varphi: X \to X$ on a compact Hausdorff space $X$ we define the entropy as follows:
Given open covering $\mathcal{U}$, $\mathcal{V}$ of $X$, we call $\mathcal{V}$ a ...

0
votes

0
answers

47
views

### Entropy maximisation of argmax over random vector

Let's say I have a vector of continuous random variables $X=[x_1, x_2, ..., x_n]$. Each one of them has its own pdf $p(x_i)$. Let us define a new random variable $i$ given by $i=\arg\max X$.
The ...

2
votes

0
answers

194
views

### Prove or disprove a mutual information inequality

I have $n$ IID Bernoulli random variables denoted by $X_1,X_2,\ldots X_n$ with parameter $p$.
I am interested in knowing if the following inequality involving mutual information holds :
$\boxed{\max_{...

2
votes

1
answer

73
views

### Mutual information between two discrete random variables

I have 2 IID random variables $X_1$ and $X_2$ with $Bern(p)$ distribution. I have another binary random variable $Y$ taking values in $\{0,1\}$.
I am interested in comparing the following 2 mutual ...

10
votes

2
answers

304
views

### The origin of the natural base in statistical mechanics

In modern treatments of statistical mechanics, the natural base is conventionally used for the Gibbs and Boltzmann entropy without careful justification. While I am aware that the properties of the ...

5
votes

1
answer

120
views

### Divergence for Bhattacharya Information Matrix

The Fisher information matrix (in the scalar parameter case) can be obtained from the Kullback-Leibler divergence by
$$g(\theta) = -\frac{\partial}{\partial \theta}\frac{\partial}{\partial \theta'}D(...

2
votes

2
answers

100
views

### Convexity of the exponential of the negative Renyi entropy

I would like to try my luck here for the following question after failing to elicit an answer to it on math.stackexchange.com.
For $r\ge -1$, the exponential of the negative Renyi entropy is defined ...

6
votes

2
answers

311
views

### 3-periodic point implies positive topological entropy

When I learn some basic ergodic theory, I encounter an interesting exercise. As we all know, 3-periodic point often means chaos. Therefore, when a continuous map has a 3-periodic point, it may have ...

2
votes

1
answer

89
views

### Boltzmann distribution

Let's say we have n points, on which the Boltzmann distribution $P = \{p_1,p_2,...,p_n\}$ is defined. Is it generally true that $\prod_{i=1}^n p_i < \prod_{i=1}^m q_i$ if $Q = \{q_1,...,q_m \}$ is ...

1
vote

0
answers

43
views

### Using maximum entropy principle for joint probability estimation

Let $X_1, \dots, X_n, Y$ be random variables, each taking values in $\{0,1\}$. Assume that we are interested in estimating, for each $v=(v_1,\dots,v_n)\in \{0,1\}^n$, the probability
$$
p(v) = P[Y=1|...

0
votes

1
answer

155
views

### Metric entropy and topological entropy

It is well known that, for a dynamical system $T$ on a metric space $(X,d)$, the variational principle connects the definition of metric entropy and topological entropy. In other words,
if
$$M(X,T) := ...

0
votes

0
answers

94
views

### Following a calculation of entropy in the first quantization scheme

I'm trying to follow the computations of example 5.1 in this paper. To begin with they have a symplectic Hilbert space $(\mathcal{K},\tau,\sigma)$, where $(\mathcal{K},\tau)$ is a separable Hilbert ...

3
votes

1
answer

88
views

### Zero entropy and the Koopman representation

Let $T$ be a measure preserving bijection of a probability space $(X,\nu)$. Consider the Koopman representation of $\mathbb{Z}$ on $L^2(X,\nu)$ given by $[z.f](x) = f(T^{-z}(x))$. The question is: can ...

2
votes

0
answers

83
views

### Continuity of the entropy of the solution of a parabolic PDE at $t=0$

Consider the following initial value problem for a parabolic PDE :
$$\begin{cases}
\textrm{div}\big(A\,\nabla u(t,x) + b(x)\, u(t,x)\big) \,=\,\partial_t u(t,x) \quad x\in\mathbb R^d\,,\ t>0 \\[4pt]...

6
votes

3
answers

802
views

### Is there an entropy proof for bounding a weighted sum of binomial coefficients?

Given a probability $p \in (0,1)$ and parameter $\alpha \in (0,1)$, is there an entropy-based proof which yields a good upper bound for the sum
$$\sum_{\ell = 0}^{\alpha n} \binom{n}{\ell}p^\ell(1-p)^{...

2
votes

2
answers

172
views

### Difference between Shannon entropy and min-entropy

I would like to construct a sequence of discrete random variable $X_2, X_3,...,X_n,...$, where $X_n \in\{0,1,2,...,n-1\}$. Given any $\epsilon \in (0,1)$, its Shannon entropy and min-entropy should ...

2
votes

1
answer

135
views

### An inequality in the optimality of Bayes' theorem

$\DeclareMathOperator\Ent{Ent}\newcommand{\prior}{\mathrm{prior}}\newcommand\Data{\mathrm{Data}}$I came across this paper on the optimality of Bayes' theorem
https://sinews.siam.org/Portals/Sinews2/...

1
vote

0
answers

49
views

### How does a computer program recognize shocks given data of a solution to a conservation law?

Conservation laws are PDEs of the form $u_t +j_x=0.$ A discontinuous solution (for $u$ and $j$) to an equation like this can be easily found. Let's suppose that we are working with a piecewise ...

0
votes

1
answer

125
views

### Trying to prove an inequality (looks similar to entropy)

I'm trying to prove the following inequality (or something similar, up to a constant factor in either side of the inequality):
$$k\cdot\sum_{i=1}^{k}x_{i}\cdot\ln\left(x_{i}\right)\geq\sum_{i=1}^{k}x_{...

0
votes

1
answer

178
views

### Obtaining the error term of binomial distribution's entropy from the differential entropy of a Gaussian distribution

It is known that the first order error term in the Shannon entropy formula for a binomial distribution is $1/n$ (for example, see the Wikipedia page Binomial distribution), where in the limit $n \to \...

14
votes

4
answers

2k
views

### Geometric interpretations of the exponential of entropy

Question:
Might there be a natural geometric interpretation of the exponential of entropy in Classical and Quantum Information theory? This question occurred to me recently via a geometric inequality ...

1
vote

0
answers

311
views

### When inequality in Mrs. Gerber's lemma is almost equality?

Let $X=x_1\ldots x_n$ be a random variable.
Assume that every $x_i$ takes values in $\{0,1\}$.
Assume also that for every $I \subseteq \{1,\ldots, n\}$ the Shannon entropy of random value $X_I$
[if $I ...

3
votes

1
answer

161
views

### What does it mean by "converges boundedly"?

On page 92 of the book Hyperbolic Conservation Laws in Continuum Physics by C. M. Dafermos, there is a theorem 4.6.1 which says
Under some assumptions, suppose a sequence of solutions $U_{\mu_k}$ to ...

6
votes

2
answers

412
views

### Definition of entropy and entropy flux of conservation laws: component-wise reasoning

Consider the conservation law
$$\DeclareMathOperator{\dvg}{\operatorname{div}}
\partial_t u(x,t) + \dvg G(u(x,t)) =0, \\
u \in U\subseteq \mathbb R^m, x\in X\subseteq \mathbb R^n, G \subseteq \mathbb ...

0
votes

1
answer

192
views

### Reference for entropy of a binomial distribution

In Wikipedia, the entropy of binomial distribution, Binomial(n,p), is written as
$\frac{1}{2} \ln (2 \pi e n p (1-p)) + O(1/n)$. Can anyone name a reference what is exactly $O(1/n)$, that is, the ...

0
votes

0
answers

85
views

### On topological entropy of continuous piecewise linear maps

Let $f$ be a continuous piecewise linear function with $k$ pieces (say the turning points are $c_1,...c_{k}$ and slopes $s_1,...,s_k$). I am curious to see if $c_1,...,c_k$ are iid uniform from [-1,1] ...

2
votes

0
answers

77
views

### A result of the covering number

Suppose $\mathcal{F} = \{f_x : x \in \mathbb{R}^d \}$ and each $f_x$ shares the same law $P$. If $\mathcal{F}$ is a class of uniformly bounded functions satisfying $L_r$-continuity, i.e. $\forall f \...

4
votes

1
answer

437
views

### Are these topological sequence entropy definition equivalent?

I am working on Möbius disjointness for models of topological dynamic systems. In that purpose, I try to understand the notion of topological entropy. We know, for a t.d.s $(X,T)$ that it is defined ...

1
vote

1
answer

82
views

### Entropy condition for quasi-linear evolution equations

Let's consider the problem:
$$
\partial_t u + \partial_x(f(u)) = 0, (x,t)\in \mathbb R \times \mathbb R^+.\\
u|_{t=0}=u_0.
$$
I have seen three formulations for the entropy condition of this equation. ...

1
vote

0
answers

57
views

### Discrete Orlicz space estimate

We consider the discrete LlogL space of sequences $x=(x_i)$ such that
$$\Vert x\Vert_{LL}:=\sum_i \vert x_i \log(x_i)\vert <\infty.$$
Let $x=(x_i)$ and $y=(y_i)$ two sequences in the above LlogL ...

2
votes

0
answers

39
views

### Maximum entropy distribution in the hyperbolic plane with given "mean" and "variance"

On the Euclidean sphere, by defining suitable analogs of the mean and variance in terms of the first circular moment, it can be shown that the von Mises distribution is the maximum entropy ...

2
votes

1
answer

222
views

### Is total variation distance of normalized sum of random variables to Gaussian monotonic decreasing?

Let $X_1, X_2, \ldots$ be independent and identically distributed random variables with mean $0$ and variance $1$ and let $S_n = (X_1 + \cdots + X_n)/\sqrt{n}$ to be their normalized sum. Define $D_n$ ...

0
votes

1
answer

113
views

### Existence of sequence of distributions

This question concerns distributions $\mu$ over the naturals $\mathbb{N}=\{1,2,\ldots\}$. For $q\ge1$, let us define the $q$th moment of entropy:
$$
H_q(\mu)=\sum_{i=1}^\infty \mu(i)|\log\mu(i)|^q,
$$
...

0
votes

1
answer

131
views

### Non-minimal system in which every point is a full entropy point

Is there a discrete topological dynamical system $(X,f)$, where $X$ is a compact metric space (with distance $d$), which is transitive but not minimal, such that $h(f)>0$ and every point is a full ...

1
vote

0
answers

33
views

### Marginal distribution of $I$-projection

I am reading this paper by Csiszar. Given a probability measure $R$ and a convex subset $\mathcal{E}$ of probability distributions, it defines ‘I-projection of R on $\mathcal{E}$’ (provided there ...

8
votes

1
answer

355
views

### Relative Entropy and p-norm

I asked this question on StackExchange but could not get any answer, therefore, I am posting it here.
I am currently reading the book "A Dynamical Approach to Random Matrix Theory". The ...

0
votes

1
answer

244
views

### Integrability of $\int \log(f(x)) f(x) dx$ for a probability density function $f$

I am looking for weak conditions when a probability density function $f$ on $\mathbb{R}^d$ has a finite integral
$$
\int_{\mathbb{R}^d} \log(f(x)) f(x) dx.
$$
Any references would be appreciated.

0
votes

0
answers

24
views

### Proof of property for Fiedland entropy

I am working with Friedland entropy and there is a proof I cannot figure out how to do.
Friedland entropy is defined for $\mathbb{Z}^k$ continuos actions $\mathcal{T}$ on a topological metric space $X$...

5
votes

3
answers

386
views

### Is there a quantum analog of Kolmogorov Complexity?

Kolmogorov Complexity (interpreted in terms of shortest program computing a string) and Shannon Entropy are quite similar.
Since there is a quantum entropy is it reasonable to ask if there is quantum ...

8
votes

1
answer

360
views

### The entropy of a partition of unity

A partition of unity can be thought of as a blurred out set partition: identify the fibers of the set partition with their indicator functions; the sum of these indicators is $1$. I would like to ...

0
votes

1
answer

127
views

### Entropy of a refinement of a partition

We consider a probability space $(X, B, \mu)$. Let $\alpha$ and $\beta$ be countable partitions of X. We suppose $\beta$ is a refinement of $\alpha$, ie that every set in $\alpha$ is a union of sets ...

6
votes

2
answers

312
views

### Shannon entropy and doubly stochastic matrices

Suppose that $A$ is a stochastic matrix. We know that if $A$ is doubly stochastic, then $H(Ap)\geq H(p)$ where $H$ is Shannon entropy and $p$ is a probability vector. Is the converse true? i.e., if $H(...

2
votes

0
answers

59
views

### Entropy of flow and time-1 map

Let $\Phi=(\phi_t)_{t\in \mathbb{R}}$ be a continuous flow on a compact metric space $X$. Let $\mu$ be a $\phi_1$-invariant measure. Then it is not hard to verify tht $\int_{0}^{1} \phi_t\mu dt$ is $\...

0
votes

0
answers

310
views

### Does additive Gaussian noise preserves the Shannon entropy ordering?

Suppose that $Z$ is a Gaussian random variable independent of $X$ and $Y$. Moreover suppose that $h(X) \geq h(Y)$, where $h(\cdot)$ is the differential Shannon entropy.
Does relation $h(X+Z) \geq h(Y+...

24
votes

3
answers

1k
views

### Is there a 'certainty' principle?

Heisenberg's uncertainty principle is a restriction on which probability distributions can describe the position and momentum of a quantum particle.
In mathematical terms it says that if $\psi\in L^2$ ...