Questions tagged [entropy]
The entropy tag has no usage guidance.
311
questions
5
votes
1
answer
409
views
Fokker-Planck: equivalence between linear spectral gap and nonlinear displacement convexity?
In a smooth, bounded and convex domain $\Omega\subset \mathbb R^d$ consider the usual linear Fokker-Planck equation with Neumann (some would say Robin) boundary conditions
\begin{equation}
\label{FP}
\...
2
votes
1
answer
107
views
Relation between multivariate estimation error and differential entropy
On page 255 of the book "Elements of information theory" by Thomas M. Cover and Joy A. Thomas, there is a theorem: For any random variable $X$ and estimator $\hat{X}$, $$E(X-\hat{X})^2 \geq \...
1
vote
0
answers
120
views
Polynomial entropy of topological dynamical systems
For a continuous mapping $\varphi: X \to X$ on a compact Hausdorff space $X$ we define the entropy as follows:
Given open covering $\mathcal{U}$, $\mathcal{V}$ of $X$, we call $\mathcal{V}$ a ...
2
votes
0
answers
247
views
Prove or disprove a mutual information inequality
I have $n$ IID Bernoulli random variables denoted by $X_1,X_2,\ldots X_n$ with parameter $p$.
I am interested in knowing if the following inequality involving mutual information holds :
$\boxed{\max_{...
2
votes
1
answer
219
views
Mutual information between two discrete random variables
I have 2 IID random variables $X_1$ and $X_2$ with $Bern(p)$ distribution. I have another binary random variable $Y$ taking values in $\{0,1\}$.
I am interested in comparing the following 2 mutual ...
10
votes
2
answers
486
views
The origin of the natural base in statistical mechanics
In modern treatments of statistical mechanics, the natural base is conventionally used for the Gibbs and Boltzmann entropy without careful justification. While I am aware that the properties of the ...
5
votes
0
answers
187
views
Divergence for Bhattacharya Information matrix
The Fisher information matrix (in the scalar parameter case) can be obtained from the Kullback-Leibler divergence by
$$g(\theta) = -\frac{\partial}{\partial \theta}\frac{\partial}{\partial \theta'}D(...
2
votes
2
answers
132
views
Convexity of the exponential of the negative Renyi entropy
I would like to try my luck here for the following question after failing to elicit an answer to it on math.stackexchange.com.
For $r\ge -1$, the exponential of the negative Renyi entropy is defined ...
6
votes
2
answers
414
views
3-periodic point implies positive topological entropy
When I learn some basic ergodic theory, I encounter an interesting exercise. As we all know, 3-periodic point often means chaos. Therefore, when a continuous map has a 3-periodic point, it may have ...
2
votes
1
answer
116
views
Boltzmann distribution
Let's say we have n points, on which the Boltzmann distribution $P = \{p_1,p_2,...,p_n\}$ is defined. Is it generally true that $\prod_{i=1}^n p_i < \prod_{i=1}^m q_i$ if $Q = \{q_1,...,q_m \}$ is ...
1
vote
0
answers
116
views
Using maximum entropy principle for joint probability estimation
Let $X_1, \dots, X_n, Y$ be random variables, each taking values in $\{0,1\}$. Assume that we are interested in estimating, for each $v=(v_1,\dots,v_n)\in \{0,1\}^n$, the probability
$$
p(v) = P[Y=1|...
1
vote
0
answers
282
views
Can a lower bound for this weakening of Goldbach's conjecture be reached?
Say a non negative integer $r$ is a primality radius of $n$ if both $n-r$ and $n+r$ are prime, and that a non negative integer $w$ is a Galois radius of $m$ if $\omega(m-w)=\omega(m+w)=1$, where $\...
1
vote
1
answer
254
views
Metric entropy and topological entropy
It is well known that, for a dynamical system $T$ on a metric space $(X,d)$, the variational principle connects the definition of metric entropy and topological entropy. In other words,
if
$$M(X,T) := ...
3
votes
1
answer
151
views
Zero entropy and the Koopman representation
Let $T$ be a measure preserving bijection of a probability space $(X,\nu)$. Consider the Koopman representation of $\mathbb{Z}$ on $L^2(X,\nu)$ given by $[z.f](x) = f(T^{-z}(x))$. The question is: can ...
2
votes
0
answers
85
views
Continuity of the entropy of the solution of a parabolic PDE at $t=0$
Consider the following initial value problem for a parabolic PDE :
$$\begin{cases}
\textrm{div}\big(A\,\nabla u(t,x) + b(x)\, u(t,x)\big) \,=\,\partial_t u(t,x) \quad x\in\mathbb R^d\,,\ t>0 \\[4pt]...
6
votes
3
answers
1k
views
Is there an entropy proof for bounding a weighted sum of binomial coefficients?
Given a probability $p \in (0,1)$ and parameter $\alpha \in (0,1)$, is there an entropy-based proof which yields a good upper bound for the sum
$$\sum_{\ell = 0}^{\alpha n} \binom{n}{\ell}p^\ell(1-p)^{...
2
votes
2
answers
624
views
Difference between Shannon entropy and min-entropy
I would like to construct a sequence of discrete random variable $X_2, X_3,...,X_n,...$, where $X_n \in\{0,1,2,...,n-1\}$. Given any $\epsilon \in (0,1)$, its Shannon entropy and min-entropy should ...
2
votes
1
answer
280
views
An inequality in the optimality of Bayes' theorem
$\DeclareMathOperator\Ent{Ent}\newcommand{\prior}{\mathrm{prior}}\newcommand\Data{\mathrm{Data}}$I came across this paper on the optimality of Bayes' theorem
https://sinews.siam.org/Portals/Sinews2/...
1
vote
0
answers
56
views
How does a computer program recognize shocks given data of a solution to a conservation law?
Conservation laws are PDEs of the form $u_t +j_x=0.$ A discontinuous solution (for $u$ and $j$) to an equation like this can be easily found. Let's suppose that we are working with a piecewise ...
0
votes
1
answer
140
views
Trying to prove an inequality (looks similar to entropy)
I'm trying to prove the following inequality (or something similar, up to a constant factor in either side of the inequality):
$$k\cdot\sum_{i=1}^{k}x_{i}\cdot\ln\left(x_{i}\right)\geq\sum_{i=1}^{k}x_{...
1
vote
1
answer
330
views
Obtaining the error term of binomial distribution's entropy from the differential entropy of a Gaussian distribution
It is known that the first order error term in the Shannon entropy formula for a binomial distribution is $1/n$ (for example, see the Wikipedia page Binomial distribution), where in the limit $n \to \...
17
votes
4
answers
2k
views
Geometric interpretations of the exponential of entropy
Question:
Might there be a natural geometric interpretation of the exponential of entropy in Classical and Quantum Information theory? This question occurred to me recently via a geometric inequality ...
1
vote
0
answers
397
views
When inequality in Mrs. Gerber's lemma is almost equality?
Let $X=x_1\ldots x_n$ be a random variable.
Assume that every $x_i$ takes values in $\{0,1\}$.
Assume also that for every $I \subseteq \{1,\ldots, n\}$ the Shannon entropy of random value $X_I$
[if $I ...
3
votes
1
answer
337
views
What does it mean by "converges boundedly"?
On page 92 of the book Hyperbolic Conservation Laws in Continuum Physics by C. M. Dafermos, there is a theorem 4.6.1 which says
Under some assumptions, suppose a sequence of solutions $U_{\mu_k}$ to ...
6
votes
2
answers
1k
views
Definition of entropy and entropy flux of conservation laws: component-wise reasoning
Consider the conservation law
$$\DeclareMathOperator{\dvg}{\operatorname{div}}
\partial_t u(x,t) + \dvg G(u(x,t)) =0, \\
u \in U\subseteq \mathbb R^m, x\in X\subseteq \mathbb R^n, G \subseteq \mathbb ...
0
votes
1
answer
832
views
Reference for entropy of a binomial distribution
In Wikipedia, the entropy of binomial distribution, Binomial(n,p), is written as
$\frac{1}{2} \ln (2 \pi e n p (1-p)) + O(1/n)$. Can anyone name a reference what is exactly $O(1/n)$, that is, the ...
2
votes
0
answers
118
views
A result of the covering number
Suppose $\mathcal{F} = \{f_x : x \in \mathbb{R}^d \}$ and each $f_x$ shares the same law $P$. If $\mathcal{F}$ is a class of uniformly bounded functions satisfying $L_r$-continuity, i.e. $\forall f \...
4
votes
1
answer
485
views
Are these topological sequence entropy definition equivalent?
I am working on Möbius disjointness for models of topological dynamic systems. In that purpose, I try to understand the notion of topological entropy. We know, for a t.d.s $(X,T)$ that it is defined ...
1
vote
1
answer
119
views
Entropy condition for quasi-linear evolution equations
Let's consider the problem:
$$
\partial_t u + \partial_x(f(u)) = 0, (x,t)\in \mathbb R \times \mathbb R^+.\\
u|_{t=0}=u_0.
$$
I have seen three formulations for the entropy condition of this equation. ...
1
vote
0
answers
60
views
Discrete Orlicz space estimate
We consider the discrete LlogL space of sequences $x=(x_i)$ such that
$$\Vert x\Vert_{LL}:=\sum_i \vert x_i \log(x_i)\vert <\infty.$$
Let $x=(x_i)$ and $y=(y_i)$ two sequences in the above LlogL ...
2
votes
0
answers
56
views
Maximum entropy distribution in the hyperbolic plane with given "mean" and "variance"
On the Euclidean sphere, by defining suitable analogs of the mean and variance in terms of the first circular moment, it can be shown that the von Mises distribution is the maximum entropy ...
2
votes
1
answer
417
views
Is total variation distance of normalized sum of random variables to Gaussian monotonic decreasing?
Let $X_1, X_2, \ldots$ be independent and identically distributed random variables with mean $0$ and variance $1$ and let $S_n = (X_1 + \cdots + X_n)/\sqrt{n}$ to be their normalized sum. Define $D_n$ ...
0
votes
1
answer
116
views
Existence of sequence of distributions
This question concerns distributions $\mu$ over the naturals $\mathbb{N}=\{1,2,\ldots\}$. For $q\ge1$, let us define the $q$th moment of entropy:
$$
H_q(\mu)=\sum_{i=1}^\infty \mu(i)|\log\mu(i)|^q,
$$
...
0
votes
1
answer
154
views
Non-minimal system in which every point is a full entropy point
Is there a discrete topological dynamical system $(X,f)$, where $X$ is a compact metric space (with distance $d$), which is transitive but not minimal, such that $h(f)>0$ and every point is a full ...
1
vote
0
answers
53
views
Marginal distribution of $I$-projection
I am reading this paper by Csiszar. Given a probability measure $R$ and a convex subset $\mathcal{E}$ of probability distributions, it defines ‘I-projection of R on $\mathcal{E}$’ (provided there ...
8
votes
1
answer
583
views
Relative Entropy and p-norm
I asked this question on StackExchange but could not get any answer, therefore, I am posting it here.
I am currently reading the book "A Dynamical Approach to Random Matrix Theory". The ...
0
votes
1
answer
476
views
Integrability of $\int \log(f(x)) f(x) dx$ for a probability density function $f$
I am looking for weak conditions when a probability density function $f$ on $\mathbb{R}^d$ has a finite integral
$$
\int_{\mathbb{R}^d} \log(f(x)) f(x) dx.
$$
Any references would be appreciated.
0
votes
0
answers
27
views
Proof of property for Fiedland entropy
I am working with Friedland entropy and there is a proof I cannot figure out how to do.
Friedland entropy is defined for $\mathbb{Z}^k$ continuos actions $\mathcal{T}$ on a topological metric space $X$...
5
votes
3
answers
476
views
Is there a quantum analog of Kolmogorov Complexity?
Kolmogorov Complexity (interpreted in terms of shortest program computing a string) and Shannon Entropy are quite similar.
Since there is a quantum entropy is it reasonable to ask if there is quantum ...
8
votes
1
answer
454
views
The entropy of a partition of unity
A partition of unity can be thought of as a blurred out set partition: identify the fibers of the set partition with their indicator functions; the sum of these indicators is $1$. I would like to ...
0
votes
1
answer
234
views
Entropy of a refinement of a partition
We consider a probability space $(X, B, \mu)$. Let $\alpha$ and $\beta$ be countable partitions of X. We suppose $\beta$ is a refinement of $\alpha$, ie that every set in $\alpha$ is a union of sets ...
6
votes
2
answers
449
views
Shannon entropy and doubly stochastic matrices
Suppose that $A$ is a stochastic matrix. We know that if $A$ is doubly stochastic, then $H(Ap)\geq H(p)$ where $H$ is Shannon entropy and $p$ is a probability vector. Is the converse true? i.e., if $H(...
2
votes
0
answers
83
views
Entropy of flow and time-1 map
Let $\Phi=(\phi_t)_{t\in \mathbb{R}}$ be a continuous flow on a compact metric space $X$. Let $\mu$ be a $\phi_1$-invariant measure. Then it is not hard to verify tht $\int_{0}^{1} \phi_t\mu dt$ is $\...
0
votes
0
answers
319
views
Does additive Gaussian noise preserves the Shannon entropy ordering?
Suppose that $Z$ is a Gaussian random variable independent of $X$ and $Y$. Moreover suppose that $h(X) \geq h(Y)$, where $h(\cdot)$ is the differential Shannon entropy.
Does relation $h(X+Z) \geq h(Y+...
24
votes
3
answers
1k
views
Is there a 'certainty' principle?
Heisenberg's uncertainty principle is a restriction on which probability distributions can describe the position and momentum of a quantum particle.
In mathematical terms it says that if $\psi\in L^2$ ...
3
votes
1
answer
245
views
Trace entropies
I'm studying relationships between trace entropy functionals and combinatorics and I'm faced with the following problem. Lets $\mathcal {D}$ be the following differential operator $1 -x\cdot \cfrac{d}{...
3
votes
3
answers
1k
views
bracketing number vs covering number
Just want to double check if the lemma on page 9 of this slides is correct:
http://www.math.leidenuniv.nl/~avdvaart/talks/09hilversum.pdf
Lemma: $N(\epsilon,\cal F,||\cdot||)\leq N_{[]}(2\epsilon,\cal ...
6
votes
2
answers
646
views
Explanation for why an ideal fluid doesn't have increasing entropy?
The equations of motion for a very simple ideal fluid (specifically a calorically perfect, monatomic, ideal gas) are \begin{align*}\dot{\rho}+\nabla \cdot (\rho u)=0 \;&\text{(mass conservation)} \...
10
votes
3
answers
762
views
Discrete entropy of the integer part of a random variable
Let $X$ be a real valued random variable. Of course, the integer part $\lfloor X \rfloor$ of $X$ is a discrete random variable taking values in $\mathbb{Z}$. We can therefore define its discrete ...
2
votes
1
answer
164
views
Conditional entropy - solve example
Given a random variable $X$ that is uniformly distributed on $[-b,b]$ and $Y=g(X)$ with
$$g(x) = \begin{cases} 0, ~~~ x\in [-c,c] \\ x, ~~~ \text{else}\end{cases}$$
Now I want to compute the ...