Skip to main content

Questions tagged [entropy]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
1 vote
0 answers
35 views

Maximize differential entropy of probability distribution without fixing second moment

(Crossposted from Mathematics Stackexchange) In Christopher Bishop's book "Pattern Recognition and Machine Learning", pages 53-54 specifically, he uses Lagrange multipliers to find the ...
Hippopotoman's user avatar
0 votes
0 answers
11 views

What are examples of $\epsilon$-extractable uniform randomness $H_{\rm ext}^\epsilon(\cal P)$?

[Renner and Wolf 2004] introduces the notion of $\epsilon$-smooth Renyi entropies as $$H_\alpha^\epsilon(P) \equiv \frac{1}{1-\alpha} \inf_{Q\in \mathcal B^\epsilon(P)}\log\left(\sum_z Q(z)^\alpha\...
glS's user avatar
  • 340
1 vote
1 answer
124 views

trying to get intuition into why Cross Entropy will always be greater or equal to the Entropy

I understand what entropy measures and cross entropy is the same except it is uses another distribution $q$ to compare it against $p.$ Is it because the log function is concave down so the predictions ...
Chris Blodgett's user avatar
0 votes
0 answers
162 views

About the monotonicity of the exponential entropy

This question was previously posted on MSE at About the monotonicity of the exponential entropy. In the paper The Unifying Frameworks of Information Measures the conditional exponential entropy (see ...
Upax's user avatar
  • 127
9 votes
1 answer
564 views

Entropy arguments used by Jean Bourgain

My question comes from understanding a probabilistic inequality in Bourgain's paper on Erdős simiarilty problem: Construction of sets of positive measure not containing an affine image of a given ...
Tutukeainie's user avatar
4 votes
1 answer
175 views

Maximum entropy probability distribution with fixed interval and variance?

What is the maximum entropy probability distribution if the support is a fixed interval (e.g. $[-1,1]$) with an already known variance? If we know the support is a fixed interval, then the maximum ...
Sarah Rune's user avatar
0 votes
0 answers
24 views

Metric entropy of mixed norm spaces with exponent-free bounds

Suppose $\mathcal{F}\subset L^p([0,1]^d)$ is a subset with the following property: The $L^q$-covering number of $\mathcal{F}$ is independent of $q$, for all $1\le q\le\infty$. An example of $\mathcal{...
chrisv's user avatar
  • 21
0 votes
1 answer
206 views

Reference request: log Sobolev inequality for uniform measure (uniform distribution over discrete set)

Suppose that $N \in \mathbb N_+$ is fixed and denote by $\mu = (\mu_0,\ldots,\mu_N)$ the uniform distribution on the set $\{0,1,\ldots,N\}$ (i.e., $\mu_n = \frac{1}{N+1}$ for each $0\leq n\leq N$). I ...
Fei Cao's user avatar
  • 710
2 votes
0 answers
105 views

Information inequality for Renyi divergences

Let $X^1 \ldots X^n$ be random variables on $\mathbb{R}^d$ with an arbitrary joint probability distribution $\mu$ on $\mathbb{R}^{n \times d}$. Let $\nu = \nu^1 \times \ldots \times \nu^n$ be a ...
MatrixGeek1234's user avatar
0 votes
0 answers
48 views

Constants in the entropy number of the Sobolev space

For a Sobolev space with $W^s(\Omega)$, where $\Omega\subset R^d$ is a compact space with smooth boundary, we know that the entropy number satisfies $e(\delta, W^s(\Omega, 1),\|\cdot\|_{L_\infty})\leq ...
NullOfMatrix's user avatar
1 vote
1 answer
170 views

Inequalities involving entropy: quantum discord and mutual information

My question is inspired by the following paper of Olivier and Żurek but for this question to be self-contained I will recall all the necessary definitions: for a quantum state $\rho$ we define the ...
truebaran's user avatar
  • 9,210
1 vote
1 answer
136 views

Is the Boltzmann entropy continuous in the supremum norm?

We define $U : [0, +\infty) \to [0, +\infty)$ by $U(0) := 0$ and $U (s) := s \log s$ for $s >0$. Then $U$ is strictly convex. Let $D$ be the set of all bounded non-negative continuous functions $\...
Akira's user avatar
  • 1,063
1 vote
0 answers
76 views

Estimating the entropy of the solution to an SDE

Forgive me for the poorly researched question. I'm currently working on a computer science project involving training a neural stochastic differential equation, and I've run into a problem while ...
user3002473's user avatar
1 vote
1 answer
119 views

Does every proximal dynamical system have zero topological entropy?

A dynamical system is proximal if $$\:\forall (x,y) \in X \times X, \: \liminf_{n \rightarrow \infty} d(f^{n}(x),f^{n}(y)) = 0 $$ (where $X$ is a compact metric space with metric $d$). Is it true that ...
Matej Moravik's user avatar
0 votes
1 answer
82 views

Can we lower bound this entropy by $\int_{\mathbb R^d} \rho^k (x) \, \mathrm d x$ and $\int_{\mathbb R^d} |x|^2\rho (x) \, \mathrm d x$?

We define $U : [0, \infty) \to [0, \infty)$ by $U(0) := 1$ and $U (s) := s \log s + (1-s)$ for $s >0$. Then $U$ is strictly convex. The minimum of $U$ is $0$ and is attained at $s=1$. Let $\mathcal ...
Akira's user avatar
  • 1,063
1 vote
2 answers
215 views

Is the Boltzmann entropy lower semi-continuous in the weak topology induced by $C_b (\mathbb R^d)$?

For Lebesgue-absolutely continuous probability measures $\rho\ll \mathcal{L}^d$ in the whole space $\mathbb{R}^d$ with finite second moments (i-e $\rho\in \mathcal{P}^2_{ac}(\mathbb{R}^d)$), let $$ \...
Akira's user avatar
  • 1,063
0 votes
0 answers
66 views

"Calculus" of metric entropy: How does metric entropy behave with respect to binary operators?

This is a general question, more of a reference request. tl;dr: Is there a "calculus" for computing metric entropy bounds? Given a function space $\mathcal{F}$, we may define its metric ...
kim341's user avatar
  • 9
3 votes
1 answer
118 views

Let $\mu : [0, T] \to \mathcal P_2^a (\mathbb R^d), t \mapsto \mu_t$ be absolutely continuous. Is $t \mapsto \mathcal H (\mu_t)$ continuous?

We endow the space $\mathcal P_2^a (\mathbb R^d)$ of absolutely continuous probability measures with finite second moment with the Wasserstein distance $W_2$. Let $\mathcal H (\mu)$ be the relative ...
Akira's user avatar
  • 1,063
1 vote
1 answer
89 views

Does convergence of Radon transforms of a sequence of probability distributions implies convergence of the distributions themselves?

Let $P_1,P_2,\ldots $ be a sequence of absolutely continuous probability measures on $\mathbb R^n$, and let $f_j:\mathbb R^n\to\mathbb R$ be their PDFs. Assume that $\operatorname{E}P_j = 0$ and $\...
Misha's user avatar
  • 13
1 vote
1 answer
139 views

Maximal entropy distribution on three variables knowing its marginals on any two

Observation 0: Given a finite set $X$, the probability distribution on $X$ with highest entropy is the uniform one. This is well known. Observation 1: Given two finite sets $X,Y$ and two probability ...
Gro-Tsen's user avatar
  • 30.6k
0 votes
0 answers
76 views

Does there exist an established name for the exponential of surprisal (e.g. the reciprocal of probability?)

There are several different names that I know of for the exponential of the entropy of which "diversity" and "perplexity" are fairly well-established. Tom Leinster has a very ...
Mike Battaglia's user avatar
3 votes
1 answer
389 views

Relative entropy equality for a sequence of Bernoulli random variables

We are given two joint probability distributions, $p$ and $q$, of $n$ Bernoulli random variables $X_1, X_2, \ldots, X_n$. We denote by $p(x_k\mid x^{k-1})$ the probability $\mathbb{P}_p(X_k=x_k\mid ...
Penelope Benenati's user avatar
3 votes
0 answers
107 views

Differential entropy of random Gibbs measure

There is a question I have been wondering about for a while, which I have thus far not been able to resolve. The problem revolves around random Gibbs measures. I am not very well-versed in the more ...
Jesse van Rhijn's user avatar
2 votes
1 answer
117 views

Morse-Hedlund\Coven-Hedlund theorem for non-Abelian groups

There is a well know theorem by Coven and Hedlund, in Sequences with minimal block growth, stating that the complexity function of an aperiodic sequence\configuration $\omega\in \mathcal{A}^{\mathbb{Z}...
Keen-ameteur's user avatar
3 votes
1 answer
194 views

Bound on an integral representing a difference of two relative entropies

Let $ f : [0,1] \to \mathbb{R} $ be a function satisfying: 1.) $ |f(x)| \leqslant a $ for some $ a < 1 $, and 2.) $ \int_0^1 f(x) {\mathrm d}x = 0 $. I would like to know whether the following ...
aleph's user avatar
  • 503
2 votes
2 answers
799 views

Defining a measure of uniformity for measurable subsets of $[0,1]^2$ w.r.t dimension $\alpha\in[0,2]$

Let $(X,d)$ be a metric space. If set $A\subseteq X$, let $H^{\alpha}$ be the $\alpha$-dimensional Hausdorff measure on $A$, where $\alpha\in[0,2]$ and $\text{dim}_{\text{H}}(A)$ is the Hausdorff ...
Arbuja's user avatar
  • 1
1 vote
1 answer
68 views

Example of finite closed cover with entropy strictly greater than topological entropy

I'm reading "Topological entropy bounds measure-theorettic entropy", by L.W. Goodwyn. enter link description here After Proposition 2, he mentions that "finite closed cover can yield ...
felcove's user avatar
  • 21
5 votes
1 answer
289 views

Does the entropy of a SDE with nondegenerate noise always increase?

Let $W$ be a standard Brownian motion, and let $X$ be the solution to the one dimensional SDE $$dX_t = \sigma(t, X_t) \, dW_t$$ with initial condition $X_0 = x_0$ a.s. for some $x_0 \in \mathbb R$. We ...
Nate River's user avatar
  • 5,435
0 votes
0 answers
60 views

How to prove that pseudo entropy and topological entropy are the same with only Markov inequality and continuity?

Let $(X,\rho)$ be a compact metric space and $f:X\to X$ a homeomorphism. We say $(x_1,\ldots,x_{n})\in X^n$ is a partial $n$ orbit if $f(x_i)=x_{i+1}$. Let $Sep_{\epsilon}(X,\rho_n)$ be the maximal ...
Bruno Seefeld's user avatar
8 votes
1 answer
280 views

Lower bound $\int_0^1 \frac{|f'(x)|^2}{f} \,\mathrm{d} x$ by $\int_0^1 |f-1|^2\, \mathrm{d} x$

Assume that $f$ is a probability density on $x \in (0,1)$, I want to obtain a bound of the following form (if it is possible at all): $$ \int_0^1 \frac{|f'|^2}{f} \,\mathrm{d} x \geq C\,\int_0^1 |f-1|^...
Fei Cao's user avatar
  • 710
2 votes
1 answer
207 views

Entropy reduction?

The following is a heuristic for the situation where a decision algorithm or a human, might solve a problem by reducing the entropy of the "search space" at every computation step $i$: Let $...
mathoverflowUser's user avatar
3 votes
1 answer
120 views

Conditions for: (local) lipschitz stability of I-projection

The following post builds on this post; I'll begin by quoting the setting. Background from Previous Question: $\newcommand\SS{P}\newcommand\TT{Q}$Call a Gaussian probability measure $\SS$ on $\mathbb{...
Math_Newbie's user avatar
1 vote
1 answer
112 views

References: error and stability estimates for information projection

$\newcommand\SS{P}\newcommand\TT{Q}$I will call a Gaussian probability measure $\SS$ on $\mathbb{R}^d$ isotropic if its covariance matrix is diagonal with non-vanishing determinant; i.e. $\Sigma_{i,i}&...
Math_Newbie's user avatar
1 vote
1 answer
242 views

Entropy upper bound for the union of uniform distributions over union-closed families

The following question is motivated by the recent breakthrough result by Justin Gilmer on the union-closed sets (aka Frankl) conjecture. Let $\mathcal{F}\subseteq\mathcal{P}(\mathbb{N})$ be a finite, ...
RaffaeleScandone's user avatar
1 vote
1 answer
147 views

Conditional differential entropy of sum of Gaussians

Is it possible to give an expression for the conditional differential entropy $h(A+B\mid C+D),$ where $A,B,C,D$ are normally distributed with known standard deviations $σ_A,\ldots,σ_D$ and where all ...
Christian Wagner's user avatar
3 votes
1 answer
367 views

A variational estimate related to the union closed set conjecture

Let $\varphi := \frac{\sqrt{5}-1}{2}$ be the golden ratio, and $H(x):=-x\log_2 x -(1-x) \log_2(1-x)$ be the binary entropy function for a Bernoulli random variable. Show that for all $\delta > 0$, ...
John Jiang's user avatar
  • 4,354
7 votes
1 answer
281 views

Why does non-decreasing entropy imply actual convergence to that max entropy distribution?

Let $X_n$ be i.i.d with finite variance. Let $\bar X_n=\frac 1n \sum_{i=1}^nX_i$. It is a famous result that the continuous/differential entropy of the normalized average is non-decreasing. $$\mathrm ...
Arrow's user avatar
  • 10.3k
10 votes
1 answer
274 views

Bibliography request: Entropy for continued fractions

Given a strictly positive real number $x$ we set $e(x)=\log(1+x)$ if $x$ is an integer and $$e(x)=\log(1+x)+\frac{1+\lbrace x\rbrace}{1+x}\left(e(1/\lbrace x\rbrace)-\log(1+\lbrace x\rbrace)\right)$$ ...
Roland Bacher's user avatar
3 votes
1 answer
203 views

Entropy of $f^{m(x)+n}$ of full shift

Let $(X,\mu,f)$ be a two-sided full shift system. Assume that there is $t \in \mathbb{N}$ such that for every $n \in \mathbb{N}$ and $x \in X$, we can define $T(x)=f^{n+m(x)}(x)$, where $m(x) \leq t; $...
Adam's user avatar
  • 1,011
2 votes
0 answers
97 views

Generalization of the min-entropy that looks at the top $n$ probabilities

The min-entropy of a random variable $X$ can often be much easier to compute than the Shannon entropy. This is because the min-entropy is simply a function of the most probable value, and sometimes, ...
Mike Battaglia's user avatar
4 votes
2 answers
278 views

Maximal entropy of integer partitions of $n$

Let $\operatorname{Part}(n)$ be the set of integer partitions of $n$. A partition $p \in \operatorname{Part}(n)$ has $k$ summands and $d$ distinct summand $n_i$, with $d \leq k$ and $d$ frequencies $...
Nicolas Couture-Grenier's user avatar
8 votes
0 answers
201 views

How to categorify entropy/perplexity?

(Migrating from math.stackexchange.com per commenter suggestion) I was reading Baez, Fritz, and Leinster's "A Characterization of Entropy in Terms of Information Loss", and wondered if, ...
GeoffChurch's user avatar
15 votes
1 answer
690 views

Information inequalities

What are the feasible $2^n$-tuples of entropies $h(S) := H(X_{i_1},\dots,X_{i_{|S|}})$ where $X_1,\dots,X_n$ are discrete random variables with some (unknown) joint probability distribution as $S=\{...
James Propp's user avatar
  • 19.5k
2 votes
0 answers
126 views

List decodability of Reed-Solomon codes beyond the Johnson bound

In a paper on a proximity test for Reed-Solomon codes the authors state an "extremely optimistical" conjecture on the list decodability of Reed-Solomon codes (over prime fields $\mathbb F_q$)...
U. Haboeck's user avatar
0 votes
0 answers
59 views

Maximize entropy under Kulback-Leibler divergence

I posed this question in math.stackexchange.com, but have not received any answer. I would like to try my luck here. In this question, it is to solve \begin{align} \max_p &-\int dy\,p(y)\ln p(y) \\...
Hans's user avatar
  • 2,229
2 votes
0 answers
131 views

Entropy of a sequence

I am reading the paper Sign Changes in Hecke Eigenvalues by Matomaki and Radziwill, and in one place they mention the following, It would be interesting to rule out the possibility of $\lambda_f(n)$ ...
Krishnarjun's user avatar
3 votes
1 answer
475 views

An inequality relating the Kullback-Leibler divergence of two discrete distributions with constant reference distribution

Suppose that $D_{KL}(p_1\parallel q)<1$ and $D_{KL}(p_2\parallel q)<1$. I'm trying to show that either $D_{KL}(p_1\parallel p_2)$ or $D_{KL}(p_2\parallel p_1)$ will have an upper bound close to ...
Harry Lorentz's user avatar
1 vote
1 answer
517 views

Covering numbers for products of functions from two spaces?

Exercise (HW1): Let $\mathcal{F}$ and $\mathcal{G}$ be classes of measurable function. Then for any probability measure $Q$ and any $1 \leq r \leq \infty$, (i) $N_{[]}\left(2 \epsilon, \mathcal{F}+\...
XYZ's user avatar
  • 79
2 votes
0 answers
253 views

Covering/Bracketing number of monotone functions on $\mathbb{R}$ with uniformly bounded derivatives

I am interested in the $\| \cdot \|_{\infty}$-norm bracketing number or covering number of some collection of distribution functions on $\mathbb{R}$. Let $\mathcal{F}$ consist of all distribution ...
masala's user avatar
  • 93
1 vote
0 answers
75 views

Finite dimensional subspaces of L^p, entropy estimates

The following follows from Proposition 9.6 in Approximation of zonoids by zonotopes, Acta Math. 162 (1989), no. 1-2, 73–141, by Bourgain, J., Lindenstrauss, J., and Milman, V. Let $X\subset L^1(\mu)$ ...
user1321324's user avatar

1
2 3 4 5
7