Skip to main content
42 votes
Accepted

Revisiting the unreasonable effectiveness of mathematics

A 2013 issue of Interdisciplinary Science Reviews was entirely devoted to this topic. One viewpoint, by Jesper Lützen, struck me: When Wigner claimed that the effectiveness of mathematics in the ...
32 votes
Accepted

Who is Mrs. Gerber?

Check out the original reference "A theorem on the entropy of certain binary sequences and applications - I" by Wyner and Ziv: https://doi.org/10.1109/TIT.1973.1055107. Footnote 2 on page ...
Sam Hopkins's user avatar
  • 23.1k
30 votes

Is there a mathematical and information theoretic explanation for this cube packing phenomenon?

• Concerning question 2, you might want to take a look at Simulation of cubical particle packing under mechanical vibration (2016). The precise effect mentioned in the 2017 paper is not considered in ...
Carlo Beenakker's user avatar
29 votes

Is there a Kolmogorov complexity proof of the prime number theorem?

Firstly, the proof given there doesn't really show that $p_n = O(n(\log n)^2)$ (at least not without further effort). Instead what it shows is that there are $n$ for which $p_n = O(n(\log n)^2)$, ...
Lucia's user avatar
  • 43.5k
26 votes
Accepted

Geometric interpretations of the exponential of entropy

With apologies for promoting my own work, there's a whole book on the mathematics of the exponentials of various entropies: Tom Leinster, Entropy and Diversity: The Axiomatic Approach. Cambridge ...
Tom Leinster's user avatar
  • 27.2k
24 votes

Revisiting the unreasonable effectiveness of mathematics

I have never seen any remotely plausible attempt at setting up a framework in which we are able to quantitatively calculate exactly how much effectiveness would be "reasonable," let alone ...
21 votes
Accepted

information-theoretic derivation of the prime number theorem

You may be interested in this arxiv paper [1], "Some information-theoretic computations related to the distribution of prime numbers", Ioannis Kontoyiannis, 2007. It discusses Chebyshev's ...
usul's user avatar
  • 4,489
21 votes
Accepted

John von Neumann's remark on entropy

An alternative version of Von Neumann's quote says "no one understands entropy very well". At the intuitive level, this makes sense, it is much harder to explain the concept of entropy to a ...
Carlo Beenakker's user avatar
21 votes

Question about information measurement for continuous random variable

The core issue is that Shannon's definition of differential entropy was a mistake - it doesn't have any of the nice properties that you would expect from the discrete case. Here are a couple other ...
Paul Siegel's user avatar
  • 28.8k
21 votes
Accepted

Information inequalities

Yes. The set of $2^n$ (or $2^n-1$ excluding the empty set) dimensional vectors formed by entropies is called the entropic region [1]. Inequalities on the entropic region not implied by the ...
Cheuk Ting Li's user avatar
20 votes

Research situation in the field of Information Geometry

I'm not sure I would say I'm an expert in information geometry. However, I worked for several years on the subject as a postdoc. As a disclaimer, this is entirely my own opinion and others may ...
Gabe K's user avatar
  • 5,394
20 votes

Is there a mathematical and information theoretic explanation for this cube packing phenomenon?

I doubt that a mathematically rigorous explanation of the phenomenon discovered in that paper exists using today's technology. While mathematical statistical mechanics is a well-developed field of ...
j.c.'s user avatar
  • 13.5k
18 votes

Question about information measurement for continuous random variable

There is actually nothing weird about this at all. What you have discovered is that for continuous quantities, information is scale-relative. Or, to put it another way, the "true" entropy of ...
The_Sympathizer's user avatar
16 votes
Accepted

How is the "conformal prediction" conformal?

Thanks for your interest. The term “conformal prediction” was suggested by Glenn Shafer, and at first I did not like it exactly for the reason that you mention: it has nothing (or very little) to do ...
Vladimir Vovk's user avatar
15 votes

Two reference requests: Pinsker's inequality and Pontryagin duality

Pinsker's inequality has many proofs. My favorites include Pollard's short but "magical" proof, https://www.cs.bgu.ac.il/~asml162/wiki.files/pollard-pinsker.pdf and surely the Proof from the Book is ...
Aryeh Kontorovich's user avatar
15 votes

Computational complexity theoretic incompleteness: is that a thing?

Consider the sentence $P(n)$ which says "This sentence has no proof shorter than $n$ characters." This sentence is true, and even has a proof - enumerate all strings of length $n$ and check ...
Sam Nead's user avatar
  • 26.6k
14 votes

Is there a mathematical and information theoretic explanation for this cube packing phenomenon?

The physical reason is that the cubically packed state has lower gravitational potential energy than the jammed random state. The overall process is analogous to annealing, although the reduction in ...
John Bentin's user avatar
  • 2,437
13 votes
Accepted

The Euler-Mascheroni constant and entropy

The earliest reference I have found for this result is Entropy and maximal spacings for random partitions (E. Slud, 1978). Theorem 2.2 states that the entropy $W_n=-\sum_{i=1}^n p_i \ln p_i$ of the ...
Carlo Beenakker's user avatar
13 votes

Who is Mrs. Gerber?

Aaron used to tell us (his colleagues at Bell Labs) that Jacob was renting a place from Mrs. Gerber while visiting Bell Labs and working on the lemma with Aaron. Mrs. Gerber interrupted them a few ...
Emina Soljanin's user avatar
13 votes
Accepted

The origin of the natural base in statistical mechanics

As Matt F. points out, we could just absorb a change of base of the logarithm into the coefficient. The reason that is not convenient in physics is that we would like the same coefficient $k$ to ...
Carlo Beenakker's user avatar
13 votes

Lower bound $\int_0^1 \frac{|f'(x)|^2}{f} \,\mathrm{d} x$ by $\int_0^1 |f-1|^2\, \mathrm{d} x$

First of all such a Csiszar–Kullback-Pinsker inequality or whatever cannot possibly be true since $x^2$ explodes faster than $x\log x$ so you can make a local adjustment so that the right-hand side is ...
Aleksei Kulikov's user avatar
12 votes

Generalisations of the Kullback-Leibler divergence for more than two distributions

The Kullback-Leibler divergence $D_{\rm KL}(Q||P)$ of two distributions $Q,P$ has been generalized to multiple distributions in various ways: [1] information radius: $R(P_1,\ldots P_k)=\frac{1}{k}\...
Carlo Beenakker's user avatar
12 votes

Entropy and total variation distance

Claim. If $\|P-Q\|\leq\varepsilon\leq\frac{1}{2}$, then $|H(P)-H(Q)| \leq H(\varepsilon) + \varepsilon\log N$. Proof. Let $\varepsilon':=\|P-Q\|$. Let $(X,Y)$ be an optimal coupling of $P$ and $Q$, ...
Algernon's user avatar
  • 1,714
12 votes

Revisiting the unreasonable effectiveness of mathematics

I'm not sure why it has not yet been pointed out that all known applications of mathematics to explain or predict phenomena in the real world only rely on a very weak part of mathematics. For example, ...
11 votes

Lower bounds on Kullback-Leibler divergence

Pinsker's inequality states that \begin{equation} \text{KL}(f|g)\ge B_P:=\|f-g\|^2/2, \end{equation} where $\|f-g\|:=\int|f-g|$ is the total variation norm of the difference between the ...
Iosif Pinelis's user avatar
11 votes

information-theoretic derivation of the prime number theorem

The following argument seems related in spirit (though it shows far less), but may be of independent interest. Let $X$, $N$, etc, be as you defined. Then $X = p_1^{E_1}\cdots p_k^{E_k}$ where the $...
Kevin K Lin's user avatar
11 votes

reverse KL-divergence: Bregman or not?

Define the KL convergence as in the Amari's paper linked by you: $$KL(x||y):=D_{KL}(x||y):=\sum(y_i-x_i+x_i\ln\frac{x_i}{y_i}).$$ Then $$KL(x||y)=F(x)-F(y)-\nabla F(y)\cdot(x-y)$$ if $F(x):=\sum(x_i\...
Iosif Pinelis's user avatar
11 votes

Revisiting the unreasonable effectiveness of mathematics

The question (and some replies) seems to be arguing something of the following form: There's no "unreasonable" effectiveness of mathematics. Of course maths is effective in physics! All we ...
11 votes
Accepted

Bipartite version of Hamming bound (two families of codewords with large Hamming distance)?

If the Hamming distance between $A$ and $B$ is at least $d$, it yields that $B$ is disjoint from the $(d-1)$-neighborhood of $A$. By isoperimetric inequality for a Boolean cube (Harper's theorem), the ...
Fedor Petrov's user avatar
10 votes

Is there an axiomatic characterization of the entropy of a continuous random variable?

$\newcommand{\Si}{\Sigma}\newcommand{\PP}{\mathscr P}\newcommand{\PPP}{\mathfrak P}$Let $X$ be a discrete random variable (r.v.) taking distinct values $x_1,x_2,\dots$, and let $p_i:=P(X = x_i)$. ...
Iosif Pinelis's user avatar

Only top scored, non community-wiki answers of a minimum length are eligible