MathOverflow is a question and answer site for professional mathematicians. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Sorry for the long title. What I mean is that for two vectors (a_1,...,a_n) and (b_1,...,b_n) with the property $a_i,b_i \geq 0 $ and $ \sum a_i =\sum b_i =1$.

If $ -\sum a_ilog(a_i) > -\sum b_ilog(b_i) $ implies $\sum a_i^2 < \sum b_i^2$ or something similar?

share|cite|improve this question
Certainly not as you ask it. If h(a)>h(b) implied |a|_2<|b|_2, then h(b)>h(a) would imply |b|_2<|a|_2 and you would get h(a)>h(b) if and only if |a|_2<|b|_2. This would mean that the entropy and the L2 norm are measuring the same thing (they're not). – Anthony Quas Jan 6 '13 at 4:07
@Anthony I am not sure, if it can be ture under this specific condiction. – gstar2002 Jan 6 '13 at 10:42
and I think for the case n = 2, it is true. – gstar2002 Jan 6 '13 at 21:51
up vote 2 down vote accepted

Of course, in general it is not true that inequality between entropies implies the same inequality between $\ell^2$ norms. Although this is true for $n=2$, it fails already for $n=3$ (as the entropy $H(p_1,p_2,p_3)$ is obviously not constant on the level curve determined by conditions $\sum p_i=1$ and $\sum p_i^2=Const$).

Nonetheless, there is a deep link between these two quantities. In order to explain it, it is better to somewhat change the viewpoint. Namely, given two probability distributions $P=(p_1,\dots,p_n)$ and $Q=(q_1,\dots,q_n)$ (for simplicity I assume that all $p_i,q_i$ are strictly positive), the corresponding Kullback-Leibler deviation is defined as $$ I(Q|P) = \sum p_i \log \frac{p_i}{q_i} \;, $$ and the so-called information energy as $$ \chi^2(Q,P) = \sum \biggl(\frac{p_i}{q_i}-1\biggr)^2 q_i \;. $$ Both these quantities measure "closedness" of $P$ and $Q$ (although they are not distances) and are monotone invariants of the pair $(Q,P)$ (they do not increase under quotient maps). If $Q=U_n$ is the uniform distribution, then these quantities are, up to linear rescaling, precisely the entropy and the $\ell^2$-norm of $P$, respectively, as $$ I(U_n|P) = \log n - H(P) $$ and $$ \chi^2(U_n,P) = n \sum p_i^2 - 1 \;. $$

Now, the link between $I$ and $\chi^2$ is provided by the fact that in naturally defined infinitesimal limits the Kullback-Leibler deviation $I$ and the information energy $\chi^2$ coincide and produce the Fisher information metric. One can read more about this in the corresponding wiki articles, in the old book Statistical decision rules and optimal inference by Čencov (AMS, 1982), or in more recent publications on information geometry.

share|cite|improve this answer
@ R W, first off, thanks a lot for the very informative reply! But I still have one more question, what do you mean by "naturally defined infinitesimal limits "? do you mean n go to infinite? – gstar2002 Jan 15 '13 at 22:11
No - infinitesimal here means that time goes to 0. The number of points remains fixed, but instead of a single distribution on $n$ points one looks at a family of distributions parametrized, say, by the interval $[0,1]$ (in other words, about a path in the space of distributions). – R W Jan 16 '13 at 0:32

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.