30
$\begingroup$

According to Claude Shannon, von Neumann gave him very useful advice on what to call his measure of information content [1]:

My greatest concern was what to call it. I thought of calling it 'information,' but the word was overly used, so I decided to call it 'uncertainty.' When I discussed it with John von Neumann, he had a better idea. Von Neumann told me, 'You should call it entropy, for two reasons. In the first place your uncertainty function has been used in statistical mechanics under that name, so it already has a name. In the second place, and more important, no one really knows what entropy really is, so in a debate you will always have the advantage.'

What I am curious about is what von Neumann meant with his last point. I find it particularly surprising given that he axiomatised what we now call the von Neumann entropy in Quantum Mechanics around two decades prior to Shannon's development of Classical Information Theory.

Might modern information theorists know of the specific difficulties he had in mind and whether these have been suitably addressed?

References:

  1. McIrvine, Edward C. and Tribus, Myron (1971). Energy and Information Scientific American 225(3): 179-190.
  2. von Neumann, John (1932). Mathematische Grundlagen der Quantenmechanik (Mathematical Foundations of Quantum Mechanics) Princeton University Press., . ISBN 978-0-691-02893-4.
  3. Shannon, Claude E. (1948). A Mathematical Theory of Communication Bell System Technical Journal 27: 379-423. doi:10.1002/j.1538-7305.1948.tb01338.x.
  4. Olivier Rioul. This is IT: A Primer on Shannon’s Entropy and Information. Séminaire Poincaré. 2018.
  5. E.T. Jaynes. Information Theory and Statistical Mechanics. The Physical Review. 1957.
  6. John A. Wheeler, 1990, “Information, physics, quantum: The search for links” in W. Zurek (ed.) Complexity, Entropy, and the Physics of Information. Redwood City, CA: Addison-Wesley.
$\endgroup$
3
  • 12
    $\begingroup$ Surely von Neumann's comment was partially a joke, so it may be misguided to try to interpret it too literally. But in thermodynamics, entropy is more mysterious than, say, temperature or pressure or volume, which all have immediately understandable physical interpretations. If Shannon had used the term "information" or "uncertainty," then some people would have had strong intuitions about what those words "should" mean, and would have complained. Using the term "entropy" would reduce the likelihood of objections based on pre-existing strong intuitions about the word. $\endgroup$ Sep 3, 2021 at 11:55
  • 2
    $\begingroup$ @TimothyChow I think that is certainly possible and it's a possibility I have considered. But, in Olivier Rioul's excellent overview of Classical Information Theory(bourbaphy.fr/rioul.pdf) he devotes an entire section to this point on page 8. This section is titled ' No One Knows What Entropy Really Is' and he brings up similar points to those made by Carlo Beenakker, specifically the mysterious connection between information theory and statistical mechanics. $\endgroup$ Sep 3, 2021 at 20:14
  • 3
    $\begingroup$ Rioul also says that when Shannon was asked about the matter years later, he did not recall von Neumann saying this. If there's some doubt about whether von Neumann actually made this statement, then that's another reason not to get too obsessed with figuring out exactly what von Neumann meant. (Of course we can still have fun debating what entropy "really is" without any reference to von Neumann.) $\endgroup$ Sep 3, 2021 at 21:31

3 Answers 3

21
$\begingroup$

An alternative version of Von Neumann's quote says "no one understands entropy very well". At the intuitive level, this makes sense, it is much harder to explain the concept of entropy to a novice than it is to explain energy.

One debate that existed at the time of Von Neumann$^1$ and is still argued upon$^2$ is whether the entropy $S=-{\rm tr}\,\rho\log\rho$ from information theory equals the physical (thermodynamic) entropy. This may or may not have been what Von Neumann was thinking about when he made the remark to Shannon, but it's one documented source of confusion.


$^1$ "In the 1950's Jaynes told Wigner that physical entropy is a measure of information and Wigner thought that was absurd, because the information one person possesses differs from that of another, whereas entropy can be measured with thermometers and calorimeters." [source]

$^2$ A Man Misunderstood: Von Neumann did not claim that his entropy corresponds to the phenomenological thermodynamic entropy (2007).

$\endgroup$
3
  • 1
    $\begingroup$ Why not tell us what the letter 𝜌 stands for. $\endgroup$ Sep 3, 2021 at 1:24
  • 4
    $\begingroup$ this would be the density matrix, as explained in en.wikipedia.org/wiki/Von_Neumann_entropy $\endgroup$ Sep 3, 2021 at 6:27
  • 1
    $\begingroup$ I remember hearing a talk by Hamming, and I think he mentioned:'calling a thing entropy doesn't mean it is entropy' (referring to information entropy) $\endgroup$
    – lalala
    Sep 3, 2021 at 11:22
5
$\begingroup$

It seems to me that it is actually the first reason put forward by von Neumann (your uncertainty function has been used in statistical mechanics under that name) that fully justifies Timothy Chow's remark concerning the second one: Surely von Neumann's comment was partially a joke, so it may be misguided to try to interpret it too literally.

Let me outline the context in which von Neumann gave his advice, and to what extent Shannon's entropy $$ \tag{H} H(p) = - \sum p_i \log p_i $$ was by that time (the late 40s) known and used in statistical mechanics.

Boltzmann gave a probabilistic interpretation of the Clausius' thermodynamic entropy as the negative logarithm of the "probability" for the system to be in a given state. I write "probability", because actually this quantity (called Permutabilitätmass, "permutability measure" by Boltzmann, "thermodynamic probability" by Planck, and now usually referred to as Boltzmann's entropy) was obtained by a certain limit procedure involving combinatorial calculations with infinitesimal regions in the phase space of a single molecule, and therefore was defined up to an unspecified additive constant. It results in the differential entropy (in modern terms) of the empirical distribution on the phase space. Gibbs further considered the integral of Boltzmann's entropy over the state space (nowadays called Gibbs' entropy).

Neither Boltzmann nor Gibbs were interested in the sums like (H) per se, and they appear in their work only as a technical tool (with $p_i$ not necessarily being probability weights). For instance, the proof of Theorem VIII in Gibbs' Elementary Principles in Statistical Mechanics amounts to establishing what is now known as Gibbs' inequality ($\equiv$ positivity of the Kullback - Leibler divergence for discrete distributions), but Gibbs feels no need to look at it as a property of the discrete entropy.

It is Planck who explicitly states and addresses the problem of the physical meaning of the limit procedure used by Boltzmann which leads him to the idea that quantization is a physical reality. I can't resist the temptation to quote his 1925 lecture The Origin and Development of the Quantum Theory where he recalls how, after having announced his "radiation formula" at the October 19, 1900 meeting of the Berlin Physical Society, he then faced the necessity to give it a conceptual interpretation:

If, however, the radiation formula should be shown to be absolutely exact, it would possess only a limited value, in the sense that it is a fortunate guess at an interpolation formula. Therefore, since it was first enunciated, I have been trying to give it a real physical meaning, and this problem led me to consider the relation between entropy and probability, along the lines of Boltzmann's ideas. After a few weeks of the most strenuous work of my life, the darkness lifted and an unexpected vista began to appear.

This vista was the first glimpse of the quantum theory. To enjoy very lucid Planck's style again (from the introduction to the second edition of The Theory of Heat Radiation, German original 1913, English translation 1914):

... the hypothesis of quanta as well as the heat theorem of Nernst may be reduced to the simple proposition that the thermodynamic probability of a physical state is a definite integral number, or, what amounts to the same thing, that the entropy of a state has a quite definite, positive value, which, as a minimum, becomes zero, while in contrast therewith the entropy may, according to the classical thermodynamics, decrease without limit to minus infinity. For the present, I would consider this proposition as the very quintessence of the hypothesis of quanta.

To put it in modern mathematical language, termodynamic entropy is a function, not a cocycle, so that not only its increments, but also its "absolute value" has a physical meaning.

The "absolute entropy" $S$ of Planck is equal to the product of "Shannon's entropy" (H) of the arising physically meaningful discrete probability distribution $(w_i)$ by the Boltzmann constant $k$ (also introduced by Planck) and the number of molecules $N$. This is very explicit formula (173) in the aforementioned Planck's book: $$ S = - kN \sum w_i \log w_i \;. $$

It is this very formula that is directly and equally explicitly quoted by von Neumann in Mathematische Grundlagen der Quantenmechanik on p. 210 when claiming that his quantum entropy is analogous to the thermodynamic one.

Needless to say, Planck's work was in the very mainstream of the 20th century physics. By the late 30s the definition (H) of the entropy of a probability distribution appears as standard in books on statistical mechanics (for instance, not only in the quoted by Shannon Tolman's book The principles of statistical mechanics, 1938, but also in Slater's Introduction to chemical physics, 1939).

For a physical discussion of entropy in classical statistical vs quantum mechanics see very instructive Chapter 14 (written by Goldstein, Lebowitz, Tumulka, and Zanghì) of the recent book Statistical Mechanics and Scientific Explanation. Its preprint is available at Gibbs and Boltzmann Entropy in Classical and Quantum Mechanics.

$\endgroup$
0
2
$\begingroup$

On the issue of whether this question has been suitably addressed, it is worth pointing out that since Von Neumann asked this important question our understanding of Entropy has advanced in a number of important ways; thanks to advances in Machine Learning and innovations in Algorithmic Information Theory led by Chaitin, Kolmogorov, Solomonoff, Leonid Levin and Shane Legg(the Chief Scientist of Deep Mind).

As I shall argue here, these innovations may eventually allow us to improve our fundamental ontology for conceptualising the Universe; notably on the subject of Occam's razor. However, it is reasonable to begin with the operational meaning of Shannon Entropy as it is conventionally understood today before proposing an equally reasonable convention which allows deeper insight.

The typical understanding of Entropy:

Today, the operational meaning of Shannon Entropy is given by the Asymptotic Equipartition Property(aka Shannon-McMillan-Breiman theorem) as explained in Edward Witten's 'Mini-Introduction to Information Theory'. In fact, an elegant generalisation of Witten's formulation may be derived using the Law of Large Numbers.

If a sequence of random variables $X_1,X_2,\ldots,X_n$ is drawn from an i.i.d. distribution $P(X)$ defined over a finite alphabet $\mathcal{A}$, then the AEP states that as $n \to \infty$:

\begin{equation} -\frac{1}{n}\log_2 P(X_1,X_2,\ldots,X_n) \rightarrow H(X) \tag{1} \end{equation}

where $H(X)$ is simply the entropy of $X$.

As an immediate corollary, we may define the existence of the typical set $\mathcal{A}_\varepsilon^n \subset \mathcal{A}^n$ which satisfies:

\begin{equation} \lim_{n \to \infty} P\big(X_{1:n} \in \mathcal{A}_\varepsilon^n \big)= P\left(\left\lvert \frac{1}{n} \sum_{i=1}^n \log_2 \frac{1}{P(X_i)} - H(X) \right\rvert \leq \varepsilon \right) = 1 \tag 2 \end{equation}

which allows us to define the typical probability:

\begin{equation} 2^{-n(H(X)+\varepsilon)} \leq P(X_{1:n} \in \mathcal{A}_\varepsilon^n) \leq 2^{-n(H(X)-\varepsilon)} \tag3 \end{equation}

Hence, we may deduce that almost all stochastic sequences are typical and robust Machine Learning amounts to efficiently compressing the structure of the typical set.

Expected Kolmogorov Complexity equals Shannon Entropy:

As an alternative to this popular and effective convention, we may consider the viewpoint that emerges from Kolmogorov's theory of Algorithmic Probability [3]. In the excellent comparative analysis on Shannon Entropy and Kolmogorov Complexity written by Peter Grünwald and Paul Vitanyí they establish that the Expected Kolmogorov Complexity of a random variable $X$ equals its Shannon Entropy up to an additive constant:

\begin{equation} \mathbb{E}[K_U(X)] = H(X) + \mathcal{O}(1) \tag{4} \end{equation}

Hence, we may deduce that the Shannon Entropy of a random variable in base-2 is the Expected Description Length of this random variable. What does this mean? If we generalise this formula using the Asymptotic Equipartition Theorem we may realise that machine learning systems that minimise the KL-Divergence are implicitly applying Occam's razor.

In fact, as a corollary of (4) we may derive Levin's Coding theorem which provides us with the operational definition for Occam's razor:

\begin{equation} -\log_2 P(X) = K_U(X) - \mathcal{O}(1) \tag{5} \end{equation}

where $X \in \{0,1\}^*$ is the prefix-free encoding of a single data point. Hence, we may interpret Shannon entropy as the intermediate representation of our epistemic uncertainty regarding singular observations.

References:

  1. Edward Witten. A Mini-Introduction To Information Theory. 2019.
  2. A. N. Kolmogorov Three approaches to the quantitative definition of information. Problems of Information and Transmission, 1(1):1--7, 1965
  3. L.A. Levin. Laws of information conservation (non-growth) and aspects of the foundation of probability theory. Problems Inform. Transmission, 10:206–210, 1974.
  4. Peter Grünwald and Paul Vitanyí. Shannon Information and Kolmogorov Complexity. Arxiv. 2004.
  5. MacKay, David J.C. (2003). Information Theory, Inference, and Learning Algorithms. Cambridge University Press. ISBN 0-521-64298-1.
$\endgroup$
4
  • $\begingroup$ Have you ever heard about the Shannon-McMillan-Breiman theorem? $\endgroup$
    – R W
    May 10, 2023 at 11:42
  • $\begingroup$ @RW as far as I know that is another name for the Asymptotic Equipartition Property. In particular, the book of David MacKay covers the connection between the AEP and Shannon Source Coding theorem. Might there be a subtle difference? $\endgroup$ May 10, 2023 at 17:22
  • $\begingroup$ The way you mention Witten creates the impression that you attribute it to him $\endgroup$
    – R W
    May 10, 2023 at 21:00
  • 1
    $\begingroup$ @RW that's a fair point. I just fixed it. $\endgroup$ May 10, 2023 at 21:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.