Questions tagged [entropy]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
1 vote
0 answers
26 views

A distribution $\pi \propto \exp(-f)$ satisfies log-Sobolev inequality, does $\exp(-af)$ also satisfy LSI?

Assume a distribution $\pi \propto e^{-f}$ satisfies log-Sobolev inequality (LSI) $$\forall \rho \in P(\mathbb{R}^n), \quad KL(\rho\| \pi) \le \frac{1}{2\lambda} I(\rho \| \pi)$$ with LSI constant $\...
user avatar
7 votes
2 answers
182 views

Does entropy of the random walk control the return probability

Given an infinite connected graph $G$ of bounded degree with vertex set $X$, let $P_x^n$ the time $n$ distribution of the simple random walk started at the vertex $x$ (so $P^n_x(y)$ is the probability ...
user avatar
  • 4,214
4 votes
1 answer
230 views

Bipartite version of Hamming bound (two families of codewords with large Hamming distance)?

Update: In light of Fedor Petrov's answer, I added an additional requirement that all strings in $A$ and $B$ have Hamming weight exactly $n/2$, which hopefully makes the question more interesting. ...
user avatar
  • 43
1 vote
1 answer
109 views

Fokker-Planck: equivalence between linear spectral gap and nonlinear displacement convexity?

In a smooth, bounded and convex domain $\Omega\subset \mathbb R^d$ consider the usual linear Fokker-Planck equation with Neumann (some would say Robin) boundary conditions \begin{equation} \label{FP} \...
user avatar
0 votes
0 answers
30 views

Relation between multivariate estimation error and differential entropy

On page 255 of the book "Elements of information theory" by Thomas M. Cover and Joy A. Thomas, there is a theorem: For any random variable $X$ and estimator $\hat{X}$, $$E(X-\hat{X})^2 \geq \...
user avatar
1 vote
0 answers
57 views

Polynomial entropy of topological dynamical systems

For a continuous mapping $\varphi: X \to X$ on a compact Hausdorff space $X$ we define the entropy as follows: Given open covering $\mathcal{U}$, $\mathcal{V}$ of $X$, we call $\mathcal{V}$ a ...
user avatar
0 votes
0 answers
47 views

Entropy maximisation of argmax over random vector

Let's say I have a vector of continuous random variables $X=[x_1, x_2, ..., x_n]$. Each one of them has its own pdf $p(x_i)$. Let us define a new random variable $i$ given by $i=\arg\max X$. The ...
user avatar
  • 123
2 votes
0 answers
194 views

Prove or disprove a mutual information inequality

I have $n$ IID Bernoulli random variables denoted by $X_1,X_2,\ldots X_n$ with parameter $p$. I am interested in knowing if the following inequality involving mutual information holds : $\boxed{\max_{...
user avatar
2 votes
1 answer
73 views

Mutual information between two discrete random variables

I have 2 IID random variables $X_1$ and $X_2$ with $Bern(p)$ distribution. I have another binary random variable $Y$ taking values in $\{0,1\}$. I am interested in comparing the following 2 mutual ...
user avatar
10 votes
2 answers
304 views

The origin of the natural base in statistical mechanics

In modern treatments of statistical mechanics, the natural base is conventionally used for the Gibbs and Boltzmann entropy without careful justification. While I am aware that the properties of the ...
user avatar
  • 3,203
5 votes
1 answer
120 views

Divergence for Bhattacharya Information Matrix

The Fisher information matrix (in the scalar parameter case) can be obtained from the Kullback-Leibler divergence by $$g(\theta) = -\frac{\partial}{\partial \theta}\frac{\partial}{\partial \theta'}D(...
user avatar
  • 769
2 votes
2 answers
100 views

Convexity of the exponential of the negative Renyi entropy

I would like to try my luck here for the following question after failing to elicit an answer to it on math.stackexchange.com. For $r\ge -1$, the exponential of the negative Renyi entropy is defined ...
user avatar
  • 2,019
6 votes
2 answers
311 views

3-periodic point implies positive topological entropy

When I learn some basic ergodic theory, I encounter an interesting exercise. As we all know, 3-periodic point often means chaos. Therefore, when a continuous map has a 3-periodic point, it may have ...
user avatar
2 votes
1 answer
89 views

Boltzmann distribution

Let's say we have n points, on which the Boltzmann distribution $P = \{p_1,p_2,...,p_n\}$ is defined. Is it generally true that $\prod_{i=1}^n p_i < \prod_{i=1}^m q_i$ if $Q = \{q_1,...,q_m \}$ is ...
user avatar
  • 21
1 vote
0 answers
43 views

Using maximum entropy principle for joint probability estimation

Let $X_1, \dots, X_n, Y$ be random variables, each taking values in $\{0,1\}$. Assume that we are interested in estimating, for each $v=(v_1,\dots,v_n)\in \{0,1\}^n$, the probability $$ p(v) = P[Y=1|...
user avatar
0 votes
1 answer
155 views

Metric entropy and topological entropy

It is well known that, for a dynamical system $T$ on a metric space $(X,d)$, the variational principle connects the definition of metric entropy and topological entropy. In other words, if $$M(X,T) := ...
user avatar
0 votes
0 answers
94 views

Following a calculation of entropy in the first quantization scheme

I'm trying to follow the computations of example 5.1 in this paper. To begin with they have a symplectic Hilbert space $(\mathcal{K},\tau,\sigma)$, where $(\mathcal{K},\tau)$ is a separable Hilbert ...
user avatar
3 votes
1 answer
88 views

Zero entropy and the Koopman representation

Let $T$ be a measure preserving bijection of a probability space $(X,\nu)$. Consider the Koopman representation of $\mathbb{Z}$ on $L^2(X,\nu)$ given by $[z.f](x) = f(T^{-z}(x))$. The question is: can ...
user avatar
  • 1,270
2 votes
0 answers
83 views

Continuity of the entropy of the solution of a parabolic PDE at $t=0$

Consider the following initial value problem for a parabolic PDE : $$\begin{cases} \textrm{div}\big(A\,\nabla u(t,x) + b(x)\, u(t,x)\big) \,=\,\partial_t u(t,x) \quad x\in\mathbb R^d\,,\ t>0 \\[4pt]...
user avatar
  • 301
6 votes
3 answers
802 views

Is there an entropy proof for bounding a weighted sum of binomial coefficients?

Given a probability $p \in (0,1)$ and parameter $\alpha \in (0,1)$, is there an entropy-based proof which yields a good upper bound for the sum $$\sum_{\ell = 0}^{\alpha n} \binom{n}{\ell}p^\ell(1-p)^{...
user avatar
  • 350
2 votes
2 answers
172 views

Difference between Shannon entropy and min-entropy

I would like to construct a sequence of discrete random variable $X_2, X_3,...,X_n,...$, where $X_n \in\{0,1,2,...,n-1\}$. Given any $\epsilon \in (0,1)$, its Shannon entropy and min-entropy should ...
user avatar
  • 23
2 votes
1 answer
135 views

An inequality in the optimality of Bayes' theorem

$\DeclareMathOperator\Ent{Ent}\newcommand{\prior}{\mathrm{prior}}\newcommand\Data{\mathrm{Data}}$I came across this paper on the optimality of Bayes' theorem https://sinews.siam.org/Portals/Sinews2/...
user avatar
  • 23
1 vote
0 answers
49 views

How does a computer program recognize shocks given data of a solution to a conservation law?

Conservation laws are PDEs of the form $u_t +j_x=0.$ A discontinuous solution (for $u$ and $j$) to an equation like this can be easily found. Let's suppose that we are working with a piecewise ...
user avatar
  • 1,457
0 votes
1 answer
125 views

Trying to prove an inequality (looks similar to entropy)

I'm trying to prove the following inequality (or something similar, up to a constant factor in either side of the inequality): $$k\cdot\sum_{i=1}^{k}x_{i}\cdot\ln\left(x_{i}\right)\geq\sum_{i=1}^{k}x_{...
user avatar
0 votes
1 answer
178 views

Obtaining the error term of binomial distribution's entropy from the differential entropy of a Gaussian distribution

It is known that the first order error term in the Shannon entropy formula for a binomial distribution is $1/n$ (for example, see the Wikipedia page Binomial distribution), where in the limit $n \to \...
user avatar
14 votes
4 answers
2k views

Geometric interpretations of the exponential of entropy

Question: Might there be a natural geometric interpretation of the exponential of entropy in Classical and Quantum Information theory? This question occurred to me recently via a geometric inequality ...
user avatar
  • 3,203
1 vote
0 answers
311 views

When inequality in Mrs. Gerber's lemma is almost equality?

Let $X=x_1\ldots x_n$ be a random variable. Assume that every $x_i$ takes values in $\{0,1\}$. Assume also that for every $I \subseteq \{1,\ldots, n\}$ the Shannon entropy of random value $X_I$ [if $I ...
user avatar
3 votes
1 answer
161 views

What does it mean by "converges boundedly"?

On page 92 of the book Hyperbolic Conservation Laws in Continuum Physics by C. M. Dafermos, there is a theorem 4.6.1 which says Under some assumptions, suppose a sequence of solutions $U_{\mu_k}$ to ...
user avatar
  • 1,457
6 votes
2 answers
412 views

Definition of entropy and entropy flux of conservation laws: component-wise reasoning

Consider the conservation law $$\DeclareMathOperator{\dvg}{\operatorname{div}} \partial_t u(x,t) + \dvg G(u(x,t)) =0, \\ u \in U\subseteq \mathbb R^m, x\in X\subseteq \mathbb R^n, G \subseteq \mathbb ...
user avatar
  • 1,457
0 votes
1 answer
192 views

Reference for entropy of a binomial distribution

In Wikipedia, the entropy of binomial distribution, Binomial(n,p), is written as $\frac{1}{2} \ln (2 \pi e n p (1-p)) + O(1/n)$. Can anyone name a reference what is exactly $O(1/n)$, that is, the ...
user avatar
0 votes
0 answers
85 views

On topological entropy of continuous piecewise linear maps

Let $f$ be a continuous piecewise linear function with $k$ pieces (say the turning points are $c_1,...c_{k}$ and slopes $s_1,...,s_k$). I am curious to see if $c_1,...,c_k$ are iid uniform from [-1,1] ...
user avatar
2 votes
0 answers
77 views

A result of the covering number

Suppose $\mathcal{F} = \{f_x : x \in \mathbb{R}^d \}$ and each $f_x$ shares the same law $P$. If $\mathcal{F}$ is a class of uniformly bounded functions satisfying $L_r$-continuity, i.e. $\forall f \...
user avatar
  • 173
4 votes
1 answer
437 views

Are these topological sequence entropy definition equivalent?

I am working on Möbius disjointness for models of topological dynamic systems. In that purpose, I try to understand the notion of topological entropy. We know, for a t.d.s $(X,T)$ that it is defined ...
user avatar
  • 137
1 vote
1 answer
82 views

Entropy condition for quasi-linear evolution equations

Let's consider the problem: $$ \partial_t u + \partial_x(f(u)) = 0, (x,t)\in \mathbb R \times \mathbb R^+.\\ u|_{t=0}=u_0. $$ I have seen three formulations for the entropy condition of this equation. ...
user avatar
  • 1,457
1 vote
0 answers
57 views

Discrete Orlicz space estimate

We consider the discrete LlogL space of sequences $x=(x_i)$ such that $$\Vert x\Vert_{LL}:=\sum_i \vert x_i \log(x_i)\vert <\infty.$$ Let $x=(x_i)$ and $y=(y_i)$ two sequences in the above LlogL ...
user avatar
2 votes
0 answers
39 views

Maximum entropy distribution in the hyperbolic plane with given "mean" and "variance"

On the Euclidean sphere, by defining suitable analogs of the mean and variance in terms of the first circular moment, it can be shown that the von Mises distribution is the maximum entropy ...
user avatar
2 votes
1 answer
222 views

Is total variation distance of normalized sum of random variables to Gaussian monotonic decreasing?

Let $X_1, X_2, \ldots$ be independent and identically distributed random variables with mean $0$ and variance $1$ and let $S_n = (X_1 + \cdots + X_n)/\sqrt{n}$ to be their normalized sum. Define $D_n$ ...
user avatar
0 votes
1 answer
113 views

Existence of sequence of distributions

This question concerns distributions $\mu$ over the naturals $\mathbb{N}=\{1,2,\ldots\}$. For $q\ge1$, let us define the $q$th moment of entropy: $$ H_q(\mu)=\sum_{i=1}^\infty \mu(i)|\log\mu(i)|^q, $$ ...
user avatar
0 votes
1 answer
131 views

Non-minimal system in which every point is a full entropy point

Is there a discrete topological dynamical system $(X,f)$, where $X$ is a compact metric space (with distance $d$), which is transitive but not minimal, such that $h(f)>0$ and every point is a full ...
user avatar
1 vote
0 answers
33 views

Marginal distribution of $I$-projection

I am reading this paper by Csiszar. Given a probability measure $R$ and a convex subset $\mathcal{E}$ of probability distributions, it defines ‘I-projection of R on $\mathcal{E}$’ (provided there ...
user avatar
8 votes
1 answer
355 views

Relative Entropy and p-norm

I asked this question on StackExchange but could not get any answer, therefore, I am posting it here. I am currently reading the book "A Dynamical Approach to Random Matrix Theory". The ...
user avatar
0 votes
1 answer
244 views

Integrability of $\int \log(f(x)) f(x) dx$ for a probability density function $f$

I am looking for weak conditions when a probability density function $f$ on $\mathbb{R}^d$ has a finite integral $$ \int_{\mathbb{R}^d} \log(f(x)) f(x) dx. $$ Any references would be appreciated.
user avatar
  • 3
0 votes
0 answers
24 views

Proof of property for Fiedland entropy

I am working with Friedland entropy and there is a proof I cannot figure out how to do. Friedland entropy is defined for $\mathbb{Z}^k$ continuos actions $\mathcal{T}$ on a topological metric space $X$...
user avatar
5 votes
3 answers
386 views

Is there a quantum analog of Kolmogorov Complexity?

Kolmogorov Complexity (interpreted in terms of shortest program computing a string) and Shannon Entropy are quite similar. Since there is a quantum entropy is it reasonable to ask if there is quantum ...
user avatar
  • 12.8k
8 votes
1 answer
360 views

The entropy of a partition of unity

A partition of unity can be thought of as a blurred out set partition: identify the fibers of the set partition with their indicator functions; the sum of these indicators is $1$. I would like to ...
user avatar
0 votes
1 answer
127 views

Entropy of a refinement of a partition

We consider a probability space $(X, B, \mu)$. Let $\alpha$ and $\beta$ be countable partitions of X. We suppose $\beta$ is a refinement of $\alpha$, ie that every set in $\alpha$ is a union of sets ...
user avatar
  • 41
6 votes
2 answers
312 views

Shannon entropy and doubly stochastic matrices

Suppose that $A$ is a stochastic matrix. We know that if $A$ is doubly stochastic, then $H(Ap)\geq H(p)$ where $H$ is Shannon entropy and $p$ is a probability vector. Is the converse true? i.e., if $H(...
user avatar
  • 109
2 votes
0 answers
59 views

Entropy of flow and time-1 map

Let $\Phi=(\phi_t)_{t\in \mathbb{R}}$ be a continuous flow on a compact metric space $X$. Let $\mu$ be a $\phi_1$-invariant measure. Then it is not hard to verify tht $\int_{0}^{1} \phi_t\mu dt$ is $\...
user avatar
0 votes
0 answers
310 views

Does additive Gaussian noise preserves the Shannon entropy ordering?

Suppose that $Z$ is a Gaussian random variable independent of $X$ and $Y$. Moreover suppose that $h(X) \geq h(Y)$, where $h(\cdot)$ is the differential Shannon entropy. Does relation $h(X+Z) \geq h(Y+...
user avatar
  • 85
24 votes
3 answers
1k views

Is there a 'certainty' principle?

Heisenberg's uncertainty principle is a restriction on which probability distributions can describe the position and momentum of a quantum particle. In mathematical terms it says that if $\psi\in L^2$ ...
user avatar

1
2 3 4 5 6