19
$\begingroup$

According to Claude Shannon, von Neumann gave him very useful advice on what to call his measure of information content [1]:

My greatest concern was what to call it. I thought of calling it 'information,' but the word was overly used, so I decided to call it 'uncertainty.' When I discussed it with John von Neumann, he had a better idea. Von Neumann told me, 'You should call it entropy, for two reasons. In the first place your uncertainty function has been used in statistical mechanics under that name, so it already has a name. In the second place, and more important, no one really knows what entropy really is, so in a debate you will always have the advantage.'

What I am curious about is what von Neumann meant with his last point. I find it particularly surprising given that he axiomatised what we now call the von Neumann entropy in Quantum Mechanics around two decades prior to Shannon's development of Classical Information Theory.

Might modern information theorists know of the specific difficulties he had in mind and whether these have been suitably addressed?

References:

  1. McIrvine, Edward C. and Tribus, Myron (1971). Energy and Information Scientific American 225(3): 179-190.
  2. von Neumann, John (1932). Mathematische Grundlagen der Quantenmechanik (Mathematical Foundations of Quantum Mechanics) Princeton University Press., . ISBN 978-0-691-02893-4.
  3. Shannon, Claude E. (1948). A Mathematical Theory of Communication Bell System Technical Journal 27: 379-423. doi:10.1002/j.1538-7305.1948.tb01338.x.
  4. Olivier Rioul. This is IT: A Primer on Shannon’s Entropy and Information. Séminaire Poincaré. 2018.
  5. E.T. Jaynes. Information Theory and Statistical Mechanics. The Physical Review. 1957.
  6. John A. Wheeler, 1990, “Information, physics, quantum: The search for links” in W. Zurek (ed.) Complexity, Entropy, and the Physics of Information. Redwood City, CA: Addison-Wesley.
$\endgroup$
3
  • 7
    $\begingroup$ Surely von Neumann's comment was partially a joke, so it may be misguided to try to interpret it too literally. But in thermodynamics, entropy is more mysterious than, say, temperature or pressure or volume, which all have immediately understandable physical interpretations. If Shannon had used the term "information" or "uncertainty," then some people would have had strong intuitions about what those words "should" mean, and would have complained. Using the term "entropy" would reduce the likelihood of objections based on pre-existing strong intuitions about the word. $\endgroup$ Sep 3, 2021 at 11:55
  • 1
    $\begingroup$ @TimothyChow I think that is certainly possible and it's a possibility I have considered. But, in Olivier Rioul's excellent overview of Classical Information Theory(bourbaphy.fr/rioul.pdf) he devotes an entire section to this point on page 8. This section is titled ' No One Knows What Entropy Really Is' and he brings up similar points to those made by Carlo Beenakker, specifically the mysterious connection between information theory and statistical mechanics. $\endgroup$ Sep 3, 2021 at 20:14
  • 3
    $\begingroup$ Rioul also says that when Shannon was asked about the matter years later, he did not recall von Neumann saying this. If there's some doubt about whether von Neumann actually made this statement, then that's another reason not to get too obsessed with figuring out exactly what von Neumann meant. (Of course we can still have fun debating what entropy "really is" without any reference to von Neumann.) $\endgroup$ Sep 3, 2021 at 21:31

2 Answers 2

17
$\begingroup$

An alternative version of Von Neumann's quote says "no one understands entropy very well". At the intuitive level, this makes sense, it is much harder to explain the concept of entropy to a novice than it is to explain energy.

One debate that existed at the time of Von Neumann$^1$ and is still argued upon$^2$ is whether the entropy $S=-{\rm tr}\,\rho\log\rho$ from information theory equals the physical (thermodynamic) entropy. This may or may not have been what Von Neumann was thinking about when he made the remark to Shannon, but it's one documented source of confusion.


$^1$ "In the 1950's Jaynes told Wigner that physical entropy is a measure of information and Wigner thought that was absurd, because the information one person possesses differs from that of another, whereas entropy can be measured with thermometers and calorimeters." [source]

$^2$ A Man Misunderstood: Von Neumann did not claim that his entropy corresponds to the phenomenological thermodynamic entropy (2007).

$\endgroup$
3
  • $\begingroup$ Why not tell us what the letter 𝜌 stands for. $\endgroup$ Sep 3, 2021 at 1:24
  • 4
    $\begingroup$ this would be the density matrix, as explained in en.wikipedia.org/wiki/Von_Neumann_entropy $\endgroup$ Sep 3, 2021 at 6:27
  • $\begingroup$ I remember hearing a talk by Hamming, and I think he mentioned:'calling a thing entropy doesn't mean it is entropy' (referring to information entropy) $\endgroup$
    – lalala
    Sep 3, 2021 at 11:22
2
$\begingroup$

It seems to me that it is actually the first reason put forward by von Neumann (your uncertainty function has been used in statistical mechanics under that name) that fully justifies Timothy Chow's remark concerning the second one: Surely von Neumann's comment was partially a joke, so it may be misguided to try to interpret it too literally.

Let me outline the context in which von Neumann gave his advice, and to what extent Shannon's entropy $$ \tag{H} H(p) = - \sum p_i \log p_i $$ was by that time (the late 40s) known and used in statistical mechanics.

Boltzmann gave a probabilistic interpretation of the Clausius' thermodynamic entropy as the negative logarithm of the "probability" for the system to be in a given state. I write "probability", because actually this quantity (called Permutabilitätmass, "permutability measure" by Boltzmann, "thermodynamic probability" by Planck, and now usually referred to as Boltzmann's entropy) was obtained by a certain limit procedure involving combinatorial calculations with infinitesimal regions in the phase space of a single molecule, and therefore was defined up to an unspecified additive constant. It results in the differential entropy (in modern terms) of the empirical distribution on the phase space. Gibbs further considered the integral of Boltzmann's entropy over the state space (nowadays called Gibbs' entropy).

Neither Boltzmann nor Gibbs were interested in the sums like (H) per se, and they appear in their work only as a technical tool (with $p_i$ not necessarily being probability weights). For instance, the proof of Theorem VIII in Gibbs' Elementary Principles in Statistical Mechanics amounts to establishing what is now known as Gibbs' inequality ($\equiv$ positivity of the Kullback - Leibler divergence for discrete distributions), but Gibbs feels no need to look at it as a property of the discrete entropy.

It is Planck who explicitly states and addresses the problem of the physical meaning of the limit procedure used by Boltzmann which leads him to the idea that quantization is a physical reality. I can't resist the temptation to quote his 1925 lecture The Origin and Development of the Quantum Theory where he recalls how, after having announced his "radiation formula" at the October 19, 1900 meeting of the Berlin Physical Society, he then faced the necessity to give it a conceptual interpretation:

If, however, the radiation formula should be shown to be absolutely exact, it would possess only a limited value, in the sense that it is a fortunate guess at an interpolation formula. Therefore, since it was first enunciated, I have been trying to give it a real physical meaning, and this problem led me to consider the relation between entropy and probability, along the lines of Boltzmann's ideas. After a few weeks of the most strenuous work of my life, the darkness lifted and an unexpected vista began to appear.

This vista was the first glimpse of the quantum theory. To enjoy very lucid Planck's style again (from the introduction to the second edition of The Theory of Heat Radiation, German original 1913, English translation 1914):

... the hypothesis of quanta as well as the heat theorem of Nernst may be reduced to the simple proposition that the thermodynamic probability of a physical state is a definite integral number, or, what amounts to the same thing, that the entropy of a state has a quite definite, positive value, which, as a minimum, becomes zero, while in contrast therewith the entropy may, according to the classical thermodynamics, decrease without limit to minus infinity. For the present, I would consider this proposition as the very quintessence of the hypothesis of quanta.

To put it in modern mathematical language, termodynamic entropy is a function, not a cocycle, so that not only its increments, but also its "absolute value" has a physical meaning.

The "absolute entropy" $S$ of Planck is equal to the product of "Shannon's entropy" (H) of the arising physically meaningful discrete probability distribution $(w_i)$ by the Boltzmann constant $k$ (also introduced by Planck) and the number of molecules $N$. This is very explicit formula (173) in the aforementioned Planck's book: $$ S = - kN \sum w_i \log w_i \;. $$

It is this very formula that is directly and equally explicitly quoted by von Neumann in Mathematische Grundlagen der Quantenmechanik on p. 210 when claiming that his quantum entropy is analogous to the thermodynamic one.

Needless to say, Planck's work was in the very mainstream of the 20th century physics. By the late 30s the definition (H) of the entropy of a probability distribution appears as standard in books on statistical mechanics (for instance, not only in the quoted by Shannon Tolman's book The principles of statistical mechanics, 1938, but also in Slater's Introduction to chemical physics, 1939).

For a physical discussion of entropy in classical statistical vs quantum mechanics see very instructive Chapter 14 (written by Goldstein, Lebowitz, Tumulka, and Zanghì) of the recent book Statistical Mechanics and Scientific Explanation. Its preprint is available at Gibbs and Boltzmann Entropy in Classical and Quantum Mechanics.

$\endgroup$
1
  • $\begingroup$ Thank you for this answer. It traces the conceptual developments from a historical perspective, which helps contextualise statements that are not so easy to interpret today. $\endgroup$ Sep 5, 2021 at 19:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.