11
$\begingroup$

For $x\in \mathbb{R}^d$, an elementary computation yields that $$\frac{d}{dp}\log \|x\|_p =\frac{1}{p^2}\sum_{i=1}^d \frac{|x_i|^p}{\|x\|_p^p}\log \frac{|x_i|^p}{\|x\|_p^p}=-\frac{1}{p^2}\operatorname{Ent}(\mu_{x,p}),$$ where $\mu_{x,p}$ is the law of a random variable taking values in $\{1,\ldots,d\}$ that takes the value $i$ with probability proportional to $|x_i|^p$ and $\operatorname{Ent}(\mu_{x,p}) = -\sum \mu_{x,p}(i) \log \mu_{x,p}(i)$ is its entropy.

I've been curious about the following question: Is there a good/conceptual reason for an entropy to appear here?

Obviously the computation is very easy, but could I have predicted this equality without any computation?

Improve this question
$\endgroup$
2
  • $\begingroup$ It may be helpful to look at the convex conjugacy of the log partition function and negative entropy (when viewed as functions of $\log |x_i|$ in this case, with entropy scaled by $1/p$ causing the corresponding "perspective transform" of the log partition function). $\endgroup$
    – usul
    Commented Jul 31, 2019 at 12:15
  • $\begingroup$ have you looked at Renyi entropy? basically it's $H_p(x) = \frac{-\log \|x\|^p_p}{p-1}$. Now if ($x$ has $\ell^1$-norm 1 and) you take the limit as $p\to 1$, you also differentiate to evaluate the limit and get the usual entropy. Not exactly the same thing as what you ask, but could be related. $\endgroup$
    – ARG
    Commented Mar 16, 2021 at 13:04

1 Answer 1

Reset to default
9
$\begingroup$

As Von Neumann said "Nobody really know what entropy is." so it is quite difficult to give a conceptual reason. However I think your calculation appears and can be interpreted in the statistical-mechanics setting (so from a physicits' point of view) as the Free Energy https://en.wikipedia.org/wiki/Helmholtz_free_energy and the canonical ensemble https://en.wikipedia.org/wiki/Canonical_ensemble

Let $E$ such that $|X|=e^{- E}$. The partition function is $Z = \sum e^{-\beta E} = \|x\|^\beta_\beta$ with $p=\beta=\frac{1}{T}$ the inverse of temperature and $F=T\log(Z)$ is the free energy. Then the physics relation $$ F = \langle E \rangle- TS $$with $S$ the entropy and $\langle E\rangle$ the mean energy and this calculation $$\langle E\rangle =\frac{1}{Z}\sum e^{-\beta E}=\partial_{\beta}\log(Z) = \beta\partial_{\beta}[\frac{1}{\beta}\log(Z)]+\frac{1}{\beta}\log(Z)= \beta \partial_{\beta}[\log(\|x\|_\beta)] +F$$ gives $$\frac{1}{\beta}S = \beta \partial_{\beta}[\log(\|x\|_\beta)$$ which is what you wanted.

Improve this answer
$\endgroup$
1
  • $\begingroup$ This is nice but I shall keep the question open for now to see what other answers are out there. I would argue that the Von Neumann quote really pertains to the metaphysical status of entropy in physics; as a mathematical object it is clearly defined, has axiomatic characterisations, and so on. $\endgroup$
    – tmh
    Commented Jul 28, 2019 at 15:41

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .