Questions tagged [entropy]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
5 votes
1 answer
409 views

Fokker-Planck: equivalence between linear spectral gap and nonlinear displacement convexity?

In a smooth, bounded and convex domain $\Omega\subset \mathbb R^d$ consider the usual linear Fokker-Planck equation with Neumann (some would say Robin) boundary conditions \begin{equation} \label{FP} \...
leo monsaingeon's user avatar
2 votes
1 answer
107 views

Relation between multivariate estimation error and differential entropy

On page 255 of the book "Elements of information theory" by Thomas M. Cover and Joy A. Thomas, there is a theorem: For any random variable $X$ and estimator $\hat{X}$, $$E(X-\hat{X})^2 \geq \...
CASIA-LH's user avatar
1 vote
0 answers
120 views

Polynomial entropy of topological dynamical systems

For a continuous mapping $\varphi: X \to X$ on a compact Hausdorff space $X$ we define the entropy as follows: Given open covering $\mathcal{U}$, $\mathcal{V}$ of $X$, we call $\mathcal{V}$ a ...
Julian Hölz's user avatar
2 votes
0 answers
247 views

Prove or disprove a mutual information inequality

I have $n$ IID Bernoulli random variables denoted by $X_1,X_2,\ldots X_n$ with parameter $p$. I am interested in knowing if the following inequality involving mutual information holds : $\boxed{\max_{...
wanderer's user avatar
2 votes
1 answer
219 views

Mutual information between two discrete random variables

I have 2 IID random variables $X_1$ and $X_2$ with $Bern(p)$ distribution. I have another binary random variable $Y$ taking values in $\{0,1\}$. I am interested in comparing the following 2 mutual ...
wanderer's user avatar
10 votes
2 answers
486 views

The origin of the natural base in statistical mechanics

In modern treatments of statistical mechanics, the natural base is conventionally used for the Gibbs and Boltzmann entropy without careful justification. While I am aware that the properties of the ...
Aidan Rocke's user avatar
  • 3,807
5 votes
0 answers
187 views

Divergence for Bhattacharya Information matrix

The Fisher information matrix (in the scalar parameter case) can be obtained from the Kullback-Leibler divergence by $$g(\theta) = -\frac{\partial}{\partial \theta}\frac{\partial}{\partial \theta'}D(...
Ashok's user avatar
  • 779
2 votes
2 answers
132 views

Convexity of the exponential of the negative Renyi entropy

I would like to try my luck here for the following question after failing to elicit an answer to it on math.stackexchange.com. For $r\ge -1$, the exponential of the negative Renyi entropy is defined ...
Hans's user avatar
  • 2,169
6 votes
2 answers
414 views

3-periodic point implies positive topological entropy

When I learn some basic ergodic theory, I encounter an interesting exercise. As we all know, 3-periodic point often means chaos. Therefore, when a continuous map has a 3-periodic point, it may have ...
user474644's user avatar
2 votes
1 answer
116 views

Boltzmann distribution

Let's say we have n points, on which the Boltzmann distribution $P = \{p_1,p_2,...,p_n\}$ is defined. Is it generally true that $\prod_{i=1}^n p_i < \prod_{i=1}^m q_i$ if $Q = \{q_1,...,q_m \}$ is ...
anhdoan's user avatar
  • 21
1 vote
0 answers
116 views

Using maximum entropy principle for joint probability estimation

Let $X_1, \dots, X_n, Y$ be random variables, each taking values in $\{0,1\}$. Assume that we are interested in estimating, for each $v=(v_1,\dots,v_n)\in \{0,1\}^n$, the probability $$ p(v) = P[Y=1|...
Bogdan Grechuk's user avatar
1 vote
0 answers
282 views

Can a lower bound for this weakening of Goldbach's conjecture be reached?

Say a non negative integer $r$ is a primality radius of $n$ if both $n-r$ and $n+r$ are prime, and that a non negative integer $w$ is a Galois radius of $m$ if $\omega(m-w)=\omega(m+w)=1$, where $\...
Sylvain JULIEN's user avatar
1 vote
1 answer
254 views

Metric entropy and topological entropy

It is well known that, for a dynamical system $T$ on a metric space $(X,d)$, the variational principle connects the definition of metric entropy and topological entropy. In other words, if $$M(X,T) := ...
Giuseppe Tenaglia's user avatar
3 votes
1 answer
151 views

Zero entropy and the Koopman representation

Let $T$ be a measure preserving bijection of a probability space $(X,\nu)$. Consider the Koopman representation of $\mathbb{Z}$ on $L^2(X,\nu)$ given by $[z.f](x) = f(T^{-z}(x))$. The question is: can ...
Vladimir's user avatar
  • 1,210
2 votes
0 answers
85 views

Continuity of the entropy of the solution of a parabolic PDE at $t=0$

Consider the following initial value problem for a parabolic PDE : $$\begin{cases} \textrm{div}\big(A\,\nabla u(t,x) + b(x)\, u(t,x)\big) \,=\,\partial_t u(t,x) \quad x\in\mathbb R^d\,,\ t>0 \\[4pt]...
tituf's user avatar
  • 311
6 votes
3 answers
1k views

Is there an entropy proof for bounding a weighted sum of binomial coefficients?

Given a probability $p \in (0,1)$ and parameter $\alpha \in (0,1)$, is there an entropy-based proof which yields a good upper bound for the sum $$\sum_{\ell = 0}^{\alpha n} \binom{n}{\ell}p^\ell(1-p)^{...
Naysh's user avatar
  • 455
2 votes
2 answers
624 views

Difference between Shannon entropy and min-entropy

I would like to construct a sequence of discrete random variable $X_2, X_3,...,X_n,...$, where $X_n \in\{0,1,2,...,n-1\}$. Given any $\epsilon \in (0,1)$, its Shannon entropy and min-entropy should ...
Luis's user avatar
  • 23
2 votes
1 answer
280 views

An inequality in the optimality of Bayes' theorem

$\DeclareMathOperator\Ent{Ent}\newcommand{\prior}{\mathrm{prior}}\newcommand\Data{\mathrm{Data}}$I came across this paper on the optimality of Bayes' theorem https://sinews.siam.org/Portals/Sinews2/...
Chp's user avatar
  • 23
1 vote
0 answers
56 views

How does a computer program recognize shocks given data of a solution to a conservation law?

Conservation laws are PDEs of the form $u_t +j_x=0.$ A discontinuous solution (for $u$ and $j$) to an equation like this can be easily found. Let's suppose that we are working with a piecewise ...
Ma Joad's user avatar
  • 1,591
0 votes
1 answer
140 views

Trying to prove an inequality (looks similar to entropy)

I'm trying to prove the following inequality (or something similar, up to a constant factor in either side of the inequality): $$k\cdot\sum_{i=1}^{k}x_{i}\cdot\ln\left(x_{i}\right)\geq\sum_{i=1}^{k}x_{...
danieljannai's user avatar
1 vote
1 answer
330 views

Obtaining the error term of binomial distribution's entropy from the differential entropy of a Gaussian distribution

It is known that the first order error term in the Shannon entropy formula for a binomial distribution is $1/n$ (for example, see the Wikipedia page Binomial distribution), where in the limit $n \to \...
user avatar
17 votes
4 answers
2k views

Geometric interpretations of the exponential of entropy

Question: Might there be a natural geometric interpretation of the exponential of entropy in Classical and Quantum Information theory? This question occurred to me recently via a geometric inequality ...
Aidan Rocke's user avatar
  • 3,807
1 vote
0 answers
397 views

When inequality in Mrs. Gerber's lemma is almost equality?

Let $X=x_1\ldots x_n$ be a random variable. Assume that every $x_i$ takes values in $\{0,1\}$. Assume also that for every $I \subseteq \{1,\ldots, n\}$ the Shannon entropy of random value $X_I$ [if $I ...
Alexey Milovanov's user avatar
3 votes
1 answer
337 views

What does it mean by "converges boundedly"?

On page 92 of the book Hyperbolic Conservation Laws in Continuum Physics by C. M. Dafermos, there is a theorem 4.6.1 which says Under some assumptions, suppose a sequence of solutions $U_{\mu_k}$ to ...
Ma Joad's user avatar
  • 1,591
6 votes
2 answers
1k views

Definition of entropy and entropy flux of conservation laws: component-wise reasoning

Consider the conservation law $$\DeclareMathOperator{\dvg}{\operatorname{div}} \partial_t u(x,t) + \dvg G(u(x,t)) =0, \\ u \in U\subseteq \mathbb R^m, x\in X\subseteq \mathbb R^n, G \subseteq \mathbb ...
Ma Joad's user avatar
  • 1,591
0 votes
1 answer
832 views

Reference for entropy of a binomial distribution

In Wikipedia, the entropy of binomial distribution, Binomial(n,p), is written as $\frac{1}{2} \ln (2 \pi e n p (1-p)) + O(1/n)$. Can anyone name a reference what is exactly $O(1/n)$, that is, the ...
user avatar
2 votes
0 answers
118 views

A result of the covering number

Suppose $\mathcal{F} = \{f_x : x \in \mathbb{R}^d \}$ and each $f_x$ shares the same law $P$. If $\mathcal{F}$ is a class of uniformly bounded functions satisfying $L_r$-continuity, i.e. $\forall f \...
香结丁's user avatar
  • 331
4 votes
1 answer
485 views

Are these topological sequence entropy definition equivalent?

I am working on Möbius disjointness for models of topological dynamic systems. In that purpose, I try to understand the notion of topological entropy. We know, for a t.d.s $(X,T)$ that it is defined ...
Jean DELI's user avatar
  • 137
1 vote
1 answer
119 views

Entropy condition for quasi-linear evolution equations

Let's consider the problem: $$ \partial_t u + \partial_x(f(u)) = 0, (x,t)\in \mathbb R \times \mathbb R^+.\\ u|_{t=0}=u_0. $$ I have seen three formulations for the entropy condition of this equation. ...
Ma Joad's user avatar
  • 1,591
1 vote
0 answers
60 views

Discrete Orlicz space estimate

We consider the discrete LlogL space of sequences $x=(x_i)$ such that $$\Vert x\Vert_{LL}:=\sum_i \vert x_i \log(x_i)\vert <\infty.$$ Let $x=(x_i)$ and $y=(y_i)$ two sequences in the above LlogL ...
The entropy method's user avatar
2 votes
0 answers
56 views

Maximum entropy distribution in the hyperbolic plane with given "mean" and "variance"

On the Euclidean sphere, by defining suitable analogs of the mean and variance in terms of the first circular moment, it can be shown that the von Mises distribution is the maximum entropy ...
popstack's user avatar
  • 265
2 votes
1 answer
417 views

Is total variation distance of normalized sum of random variables to Gaussian monotonic decreasing?

Let $X_1, X_2, \ldots$ be independent and identically distributed random variables with mean $0$ and variance $1$ and let $S_n = (X_1 + \cdots + X_n)/\sqrt{n}$ to be their normalized sum. Define $D_n$ ...
Sandeep Silwal's user avatar
0 votes
1 answer
116 views

Existence of sequence of distributions

This question concerns distributions $\mu$ over the naturals $\mathbb{N}=\{1,2,\ldots\}$. For $q\ge1$, let us define the $q$th moment of entropy: $$ H_q(\mu)=\sum_{i=1}^\infty \mu(i)|\log\mu(i)|^q, $$ ...
Aryeh Kontorovich's user avatar
0 votes
1 answer
154 views

Non-minimal system in which every point is a full entropy point

Is there a discrete topological dynamical system $(X,f)$, where $X$ is a compact metric space (with distance $d$), which is transitive but not minimal, such that $h(f)>0$ and every point is a full ...
Alessandro Della Corte's user avatar
1 vote
0 answers
53 views

Marginal distribution of $I$-projection

I am reading this paper by Csiszar. Given a probability measure $R$ and a convex subset $\mathcal{E}$ of probability distributions, it defines ‘I-projection of R on $\mathcal{E}$’ (provided there ...
Raghav's user avatar
  • 361
8 votes
1 answer
583 views

Relative Entropy and p-norm

I asked this question on StackExchange but could not get any answer, therefore, I am posting it here. I am currently reading the book "A Dynamical Approach to Random Matrix Theory". The ...
Raghav's user avatar
  • 361
0 votes
1 answer
476 views

Integrability of $\int \log(f(x)) f(x) dx$ for a probability density function $f$

I am looking for weak conditions when a probability density function $f$ on $\mathbb{R}^d$ has a finite integral $$ \int_{\mathbb{R}^d} \log(f(x)) f(x) dx. $$ Any references would be appreciated.
Austin's user avatar
  • 3
0 votes
0 answers
27 views

Proof of property for Fiedland entropy

I am working with Friedland entropy and there is a proof I cannot figure out how to do. Friedland entropy is defined for $\mathbb{Z}^k$ continuos actions $\mathcal{T}$ on a topological metric space $X$...
user502940's user avatar
5 votes
3 answers
476 views

Is there a quantum analog of Kolmogorov Complexity?

Kolmogorov Complexity (interpreted in terms of shortest program computing a string) and Shannon Entropy are quite similar. Since there is a quantum entropy is it reasonable to ask if there is quantum ...
Turbo's user avatar
  • 13.6k
8 votes
1 answer
454 views

The entropy of a partition of unity

A partition of unity can be thought of as a blurred out set partition: identify the fibers of the set partition with their indicator functions; the sum of these indicators is $1$. I would like to ...
Yakov Shklarov's user avatar
0 votes
1 answer
234 views

Entropy of a refinement of a partition

We consider a probability space $(X, B, \mu)$. Let $\alpha$ and $\beta$ be countable partitions of X. We suppose $\beta$ is a refinement of $\alpha$, ie that every set in $\alpha$ is a union of sets ...
catbow's user avatar
  • 41
6 votes
2 answers
449 views

Shannon entropy and doubly stochastic matrices

Suppose that $A$ is a stochastic matrix. We know that if $A$ is doubly stochastic, then $H(Ap)\geq H(p)$ where $H$ is Shannon entropy and $p$ is a probability vector. Is the converse true? i.e., if $H(...
Aram's user avatar
  • 109
2 votes
0 answers
83 views

Entropy of flow and time-1 map

Let $\Phi=(\phi_t)_{t\in \mathbb{R}}$ be a continuous flow on a compact metric space $X$. Let $\mu$ be a $\phi_1$-invariant measure. Then it is not hard to verify tht $\int_{0}^{1} \phi_t\mu dt$ is $\...
user119197's user avatar
0 votes
0 answers
319 views

Does additive Gaussian noise preserves the Shannon entropy ordering?

Suppose that $Z$ is a Gaussian random variable independent of $X$ and $Y$. Moreover suppose that $h(X) \geq h(Y)$, where $h(\cdot)$ is the differential Shannon entropy. Does relation $h(X+Z) \geq h(Y+...
Mini's user avatar
  • 85
24 votes
3 answers
1k views

Is there a 'certainty' principle?

Heisenberg's uncertainty principle is a restriction on which probability distributions can describe the position and momentum of a quantum particle. In mathematical terms it says that if $\psi\in L^2$ ...
Oscar Cunningham's user avatar
3 votes
1 answer
245 views

Trace entropies

I'm studying relationships between trace entropy functionals and combinatorics and I'm faced with the following problem. Lets $\mathcal {D}$ be the following differential operator $1 -x\cdot \cfrac{d}{...
Gianfranco's user avatar
3 votes
3 answers
1k views

bracketing number vs covering number

Just want to double check if the lemma on page 9 of this slides is correct: http://www.math.leidenuniv.nl/~avdvaart/talks/09hilversum.pdf Lemma: $N(\epsilon,\cal F,||\cdot||)\leq N_{[]}(2\epsilon,\cal ...
Hepdrey's user avatar
  • 92
6 votes
2 answers
646 views

Explanation for why an ideal fluid doesn't have increasing entropy?

The equations of motion for a very simple ideal fluid (specifically a calorically perfect, monatomic, ideal gas) are \begin{align*}\dot{\rho}+\nabla \cdot (\rho u)=0 \;&\text{(mass conservation)} \...
vmist's user avatar
  • 879
10 votes
3 answers
762 views

Discrete entropy of the integer part of a random variable

Let $X$ be a real valued random variable. Of course, the integer part $\lfloor X \rfloor$ of $X$ is a discrete random variable taking values in $\mathbb{Z}$. We can therefore define its discrete ...
Goulifet's user avatar
  • 2,174
2 votes
1 answer
164 views

Conditional entropy - solve example

Given a random variable $X$ that is uniformly distributed on $[-b,b]$ and $Y=g(X)$ with $$g(x) = \begin{cases} 0, ~~~ x\in [-c,c] \\ x, ~~~ \text{else}\end{cases}$$ Now I want to compute the ...
Phobos's user avatar
  • 131

1
2
3 4 5
7