42
votes

Accepted

### Revisiting the unreasonable effectiveness of mathematics

A 2013 issue of Interdisciplinary Science Reviews was entirely devoted to this topic. One viewpoint, by Jesper Lützen, struck me:
When Wigner claimed that the effectiveness of mathematics in the
...

Community wiki

32
votes

Accepted

### Who is Mrs. Gerber?

Check out the original reference "A theorem on the entropy of certain binary sequences and applications - I" by Wyner and Ziv: https://doi.org/10.1109/TIT.1973.1055107. Footnote 2 on page ...

30
votes

### Is there a mathematical and information theoretic explanation for this cube packing phenomenon?

• Concerning question 2, you might want to take a look at Simulation of cubical particle packing under mechanical vibration (2016). The precise effect mentioned in the 2017 paper is not considered in ...

29
votes

### Is there a Kolmogorov complexity proof of the prime number theorem?

Firstly, the proof given there doesn't really show that $p_n = O(n(\log n)^2)$
(at least not without further effort). Instead what it shows is that there are
$n$ for which $p_n = O(n(\log n)^2)$, ...

26
votes

Accepted

### Geometric interpretations of the exponential of entropy

With apologies for promoting my own work, there's a whole book on the
mathematics of the exponentials of various entropies:
Tom Leinster, Entropy and Diversity: The Axiomatic
Approach. Cambridge ...

24
votes

### Revisiting the unreasonable effectiveness of mathematics

I have never seen any remotely plausible attempt at setting up a framework in which we are able to quantitatively calculate exactly how much effectiveness would be "reasonable," let alone ...

Community wiki

21
votes

Accepted

### information-theoretic derivation of the prime number theorem

You may be interested in this arxiv paper [1], "Some information-theoretic computations related to the distribution of prime numbers", Ioannis Kontoyiannis, 2007.
It discusses Chebyshev's ...

21
votes

Accepted

### John von Neumann's remark on entropy

An alternative version of Von Neumann's quote says "no one understands entropy very well". At the intuitive level, this makes sense, it is much harder to explain the concept of entropy to a ...

21
votes

### Question about information measurement for continuous random variable

The core issue is that Shannon's definition of differential entropy was a mistake - it doesn't have any of the nice properties that you would expect from the discrete case. Here are a couple other ...

21
votes

Accepted

### Information inequalities

Yes. The set of $2^n$ (or $2^n-1$ excluding the empty set) dimensional vectors formed by entropies is called the entropic region [1]. Inequalities on the entropic region not implied by the ...

20
votes

### Research situation in the field of Information Geometry

I'm not sure I would say I'm an expert in information geometry. However, I worked for several years on the subject as a postdoc. As a disclaimer, this is entirely my own opinion and others may ...

20
votes

### Is there a mathematical and information theoretic explanation for this cube packing phenomenon?

I doubt that a mathematically rigorous explanation of the phenomenon discovered in that paper exists using today's technology. While mathematical statistical mechanics is a well-developed field of ...

18
votes

### Question about information measurement for continuous random variable

There is actually nothing weird about this at all. What you have discovered is that for continuous quantities, information is scale-relative. Or, to put it another way, the "true" entropy of ...

16
votes

Accepted

### How is the "conformal prediction" conformal?

Thanks for your interest. The term “conformal prediction” was suggested by Glenn Shafer, and at first I did not like it exactly for the reason that you mention: it has nothing (or very little) to do ...

15
votes

### Two reference requests: Pinsker's inequality and Pontryagin duality

Pinsker's inequality has many proofs. My favorites include Pollard's short but "magical" proof,
https://www.cs.bgu.ac.il/~asml162/wiki.files/pollard-pinsker.pdf
and surely the Proof from the Book is ...

15
votes

### Computational complexity theoretic incompleteness: is that a thing?

Consider the sentence $P(n)$ which says "This sentence has no proof shorter than $n$ characters." This sentence is true, and even has a proof - enumerate all strings of length $n$ and check ...

14
votes

### Is there a mathematical and information theoretic explanation for this cube packing phenomenon?

The physical reason is that the cubically packed state has lower gravitational potential energy than the jammed random state. The overall process is analogous to annealing, although the reduction in ...

13
votes

Accepted

### The Euler-Mascheroni constant and entropy

The earliest reference I have found for this result is Entropy and maximal spacings for random partitions (E. Slud, 1978).
Theorem 2.2 states that the entropy $W_n=-\sum_{i=1}^n p_i \ln p_i$ of the ...

13
votes

### Who is Mrs. Gerber?

Aaron used to tell us (his colleagues at Bell Labs) that Jacob was renting a place from Mrs. Gerber while visiting Bell Labs and working on the lemma with Aaron. Mrs. Gerber interrupted them a few ...

13
votes

Accepted

### The origin of the natural base in statistical mechanics

As Matt F. points out, we could just absorb a change of base of the logarithm into the coefficient. The reason that is not convenient in physics is that we would like the same coefficient $k$ to ...

13
votes

### Lower bound $\int_0^1 \frac{|f'(x)|^2}{f} \,\mathrm{d} x$ by $\int_0^1 |f-1|^2\, \mathrm{d} x$

First of all such a Csiszar–Kullback-Pinsker inequality or whatever cannot possibly be true since $x^2$ explodes faster than $x\log x$ so you can make a local adjustment so that the right-hand side is ...

12
votes

### Generalisations of the Kullback-Leibler divergence for more than two distributions

The Kullback-Leibler divergence $D_{\rm KL}(Q||P)$ of two distributions $Q,P$ has been generalized to multiple distributions in various ways:
[1] information radius: $R(P_1,\ldots P_k)=\frac{1}{k}\...

12
votes

### Entropy and total variation distance

Claim. If $\|P-Q\|\leq\varepsilon\leq\frac{1}{2}$, then $|H(P)-H(Q)| \leq H(\varepsilon) + \varepsilon\log N$.
Proof.
Let $\varepsilon':=\|P-Q\|$.
Let $(X,Y)$ be an optimal coupling of $P$ and $Q$, ...

12
votes

### Revisiting the unreasonable effectiveness of mathematics

I'm not sure why it has not yet been pointed out that all known applications of mathematics to explain or predict phenomena in the real world only rely on a very weak part of mathematics. For example, ...

Community wiki

11
votes

### Lower bounds on Kullback-Leibler divergence

Pinsker's inequality states that
\begin{equation}
\text{KL}(f|g)\ge B_P:=\|f-g\|^2/2,
\end{equation}
where $\|f-g\|:=\int|f-g|$ is the total variation norm of the difference between the ...

11
votes

### information-theoretic derivation of the prime number theorem

The following argument seems related in spirit (though it shows far less), but may be of independent interest. Let $X$, $N$, etc, be as you defined. Then $X = p_1^{E_1}\cdots p_k^{E_k}$ where the $...

11
votes

### reverse KL-divergence: Bregman or not?

Define the KL convergence as in the Amari's paper linked by you:
$$KL(x||y):=D_{KL}(x||y):=\sum(y_i-x_i+x_i\ln\frac{x_i}{y_i}).$$
Then
$$KL(x||y)=F(x)-F(y)-\nabla F(y)\cdot(x-y)$$
if $F(x):=\sum(x_i\...

11
votes

### Revisiting the unreasonable effectiveness of mathematics

The question (and some replies) seems to be arguing something of the following form:
There's no "unreasonable" effectiveness of mathematics. Of course maths is effective in physics! All we ...

Community wiki

11
votes

Accepted

### Bipartite version of Hamming bound (two families of codewords with large Hamming distance)?

If the Hamming distance between $A$ and $B$ is at least $d$, it yields that $B$ is disjoint from the $(d-1)$-neighborhood of $A$. By isoperimetric inequality for a Boolean cube (Harper's theorem), the ...

10
votes

### Is there an axiomatic characterization of the entropy of a continuous random variable?

$\newcommand{\Si}{\Sigma}\newcommand{\PP}{\mathscr P}\newcommand{\PPP}{\mathfrak P}$Let $X$ be a discrete random variable (r.v.) taking distinct values $x_1,x_2,\dots$, and let $p_i:=P(X = x_i)$. ...

Only top scored, non community-wiki answers of a minimum length are eligible

#### Related Tags

it.information-theory × 615pr.probability × 253

st.statistics × 105

entropy × 93

coding-theory × 84

co.combinatorics × 67

probability-distributions × 57

reference-request × 50

inequalities × 37

information-geometry × 30

computer-science × 24

linear-algebra × 23

graph-theory × 23

measure-theory × 19

algorithms × 18

nt.number-theory × 17

stochastic-processes × 17

quantum-mechanics × 17

computational-complexity × 15

real-analysis × 14

mg.metric-geometry × 14

mp.mathematical-physics × 14

computability-theory × 13

markov-chains × 12

fa.functional-analysis × 11