John von Neumann's remark on entropy According to Claude Shannon, von Neumann gave him very useful advice on what to call his measure of information content [1]:

My greatest concern was what to call it. I thought of calling it 'information,' but the word was overly used, so I decided to call it 'uncertainty.' When I discussed it with John von Neumann, he had a better idea. Von Neumann told me, 'You should call it entropy, for two reasons. In the first place your uncertainty function has been used in statistical mechanics under that name, so it already has a name. In the second place, and more important, no one really knows what entropy really is, so in a debate you will always have the advantage.'

What I am curious about is what von Neumann meant with his last point. I find it particularly surprising given that he axiomatised what we now call the von Neumann entropy in Quantum Mechanics around two decades prior to Shannon's development of Classical Information Theory.
Might modern information theorists know of the specific difficulties he had in mind and whether these have been suitably addressed?
References:

*

*McIrvine, Edward C. and Tribus, Myron (1971). Energy and Information Scientific American 225(3): 179-190.

*von Neumann, John (1932). Mathematische Grundlagen der Quantenmechanik (Mathematical Foundations of Quantum Mechanics) Princeton University Press., . ISBN 978-0-691-02893-4.

*Shannon, Claude E. (1948). A Mathematical Theory of Communication Bell System Technical Journal 27: 379-423. doi:10.1002/j.1538-7305.1948.tb01338.x.

*Olivier Rioul. This is IT: A Primer on Shannon’s Entropy and Information. Séminaire Poincaré. 2018.

*E.T. Jaynes. Information Theory and Statistical Mechanics. The Physical Review. 1957.

*John A. Wheeler, 1990, “Information, physics, quantum: The search for links” in W. Zurek (ed.) Complexity, Entropy, and the Physics of Information. Redwood City, CA: Addison-Wesley.

 A: It seems to me that it is actually the first reason put forward by von Neumann (your uncertainty function has been used in statistical mechanics under that name) that fully justifies Timothy Chow's remark concerning the second one: Surely von Neumann's comment was partially a joke, so it may be misguided to try to interpret it too literally.
Let me outline the context in which von Neumann gave his advice, and to what extent Shannon's entropy
$$
\tag{H}
H(p) = - \sum p_i \log p_i
$$
was by that time (the late 40s) known and used in statistical mechanics.
Boltzmann gave a probabilistic interpretation of the Clausius' thermodynamic entropy as the negative logarithm of the "probability" for the system to be in a given state. I write "probability", because actually this quantity (called Permutabilitätmass, "permutability measure" by Boltzmann, "thermodynamic probability" by Planck, and now usually referred to as Boltzmann's entropy) was obtained by a certain limit procedure involving combinatorial calculations with infinitesimal regions in the phase space of a single molecule, and therefore was defined up to an unspecified additive constant. It results in the differential entropy (in modern terms) of the empirical distribution on the phase space. Gibbs further considered the integral of Boltzmann's entropy over the state space (nowadays called Gibbs' entropy).
Neither Boltzmann nor Gibbs were interested in the sums like (H) per se, and they appear in their work only as a technical tool (with $p_i$ not necessarily being probability weights). For instance, the proof of Theorem VIII in Gibbs' Elementary Principles in Statistical Mechanics amounts to establishing what is now known as Gibbs' inequality ($\equiv$ positivity of the Kullback - Leibler divergence for discrete distributions), but Gibbs feels no need to look at it as a property of the discrete entropy.
It is Planck who explicitly states and addresses the problem of the physical meaning of the limit procedure used by Boltzmann which leads him to the idea that quantization is a physical reality. I can't resist the temptation to quote his 1925 lecture The Origin and Development of the Quantum Theory where he recalls how, after having announced his "radiation formula" at the October 19, 1900 meeting of the Berlin Physical Society, he then faced the necessity to give it a conceptual interpretation:

If, however, the radiation formula should be shown to be absolutely exact, it would possess only a limited value, in the sense that it is a fortunate guess at an interpolation formula. Therefore, since it was first enunciated, I have been trying to give it a real physical meaning, and this problem led me to consider the relation between entropy and probability, along the lines of Boltzmann's ideas. After a few weeks of the most strenuous work of my life, the darkness lifted and an unexpected vista began to appear.

This vista was the first glimpse of the quantum theory. To enjoy very lucid Planck's style again
(from the introduction to the second edition of The Theory of Heat Radiation, German original 1913, English translation 1914):

... the hypothesis of quanta as well as the heat
theorem of Nernst may be reduced to the simple proposition that the thermodynamic probability of a physical state is a definite integral number, or, what amounts to the same thing, that the entropy
of a state has a quite definite, positive value, which, as a minimum, becomes zero, while in contrast therewith the entropy may, according
to the classical thermodynamics, decrease without limit to minus infinity. For the present, I would consider this proposition as the very quintessence of the hypothesis of quanta.

To put it in modern mathematical language, termodynamic entropy is a function, not a cocycle, so that not only its increments, but also its "absolute value" has a physical meaning.
The "absolute entropy" $S$ of Planck is equal to the product of "Shannon's entropy" (H) of the arising physically meaningful discrete probability distribution $(w_i)$ by the Boltzmann constant $k$ (also introduced by Planck) and the number of molecules $N$. This is very explicit formula (173) in the aforementioned Planck's book:
$$
S = - kN \sum w_i \log w_i \;.
$$
It is this very formula that is directly and equally explicitly quoted by von Neumann in Mathematische Grundlagen der Quantenmechanik on p. 210 when claiming that his quantum entropy is analogous to the thermodynamic one.
Needless to say, Planck's work was in the very mainstream of the 20th century physics. By the late 30s the definition (H) of the entropy of a probability distribution appears as standard in books on statistical mechanics (for instance, not only in the quoted by Shannon Tolman's book The principles of statistical mechanics, 1938, but also in Slater's Introduction to chemical physics, 1939).
For a physical discussion of entropy in classical statistical vs quantum mechanics see very instructive Chapter 14 (written by Goldstein, Lebowitz, Tumulka, and Zanghì) of the recent book Statistical Mechanics and Scientific Explanation. Its preprint is available at Gibbs and Boltzmann Entropy in Classical and Quantum Mechanics.
A: An alternative version of Von Neumann's quote says "no one understands entropy very well". At the intuitive level, this makes sense, it is much harder to explain the concept of entropy to a novice than it is to explain energy.
One debate that existed at the time of Von Neumann$^1$ and is still argued upon$^2$ is whether the entropy $S=-{\rm tr}\,\rho\log\rho$ from information theory equals the physical (thermodynamic) entropy. This may or may not have been what Von Neumann was thinking about when he made the remark to Shannon, but it's one documented source of confusion.

$^1$ "In the 1950's Jaynes told Wigner that physical entropy is a measure of information and Wigner thought that was absurd, because the information one person possesses differs from that of another, whereas entropy can be measured with thermometers and calorimeters." [source]
$^2$ A Man Misunderstood: Von Neumann did not claim that his entropy corresponds to the phenomenological thermodynamic entropy (2007).
