Let $G$ be a locally compact group. A measure $\mu$ is the right-Haar measure on $G$ if for every $g\in G$ and $E\subseteq G$ Borel set $\mu(Eg)=\mu(E)$. It is known that every locally compact group has a unique right-Haar measure (up to a multiplicative constant). Similarly one can define the left-Haar measure as being invariant for left shifts. However, in the case $G$ is compact this distinction is irrelevant, since both left and right Haar measures are equivalent up to a constant (note that $G$ is not nececssarily commutative, so being left and right Haar measures are in fact two different properties).

I am interested on a natural notion of entropy for measures on compact groups. In this case I expect the Haar measure to maximize the entropy. Is there an analog of information theory developed in this framework? I am particularly interested in the case where

I am particularly interested on the case where $G=O(n)$, the group of orthogonal transformations of $\mathbb{R}^n$.

Any references or insights are greatly welcome!

---------------------------- Update ------------------------------

Due to a request, I will further explain. Before I start, for notation and definitions I refer to Cover & Thomas book (Elements of Information Theory).

My question is very much related to Intrinsic significance of differential entropy: in the case of real random variables, differential entropy is troublesome due to the fact that there is no canonical reference space (in the discrete case, we are always working with r.v.'s over $\{1,\ldots,n\}$), so for example differential entropy is not invariant under re-scalings, since $$ h(a X) = h(X) + \log(a), $$ as opposed to the discrete case where $h(aX)=h(X)$.

However, in the case your r.v.'s are defined over a compact group $G$ (we may restrict here to the ones that are absolutely continuous w.r.t. Haar), the re-scaling problem disappears, as we have a reference space $G$. I wonder if having this reference space helps at all in terms of having information-theoretic identities/inequalities similar to the discrete case (e.g., Data Processing Inequality).

Finally, from what I have been reading lately, it seems that this is hopeless: just the fact of having an infinite measure space seems to break down all nice properties of the discrete entropy. However, I want to keep this question open to further comments from people with more acquaintance with information theory.

  • 1
    $\begingroup$ In a compact grouple the distinction between left invariant and right invariant is not meaningless but irrelevant or something —compactness does not make the distinction lose its meaning! $\endgroup$ – Mariano Suárez-Álvarez Apr 18 '14 at 19:26
  • 1
    $\begingroup$ Absolutely! I will rephrase it. Thank you $\endgroup$ – Cristóbal Guzmán Apr 18 '14 at 21:04
  • 1
    $\begingroup$ entropy is defined by a dynamical system, not just a space. Which dynamical system are you considering here? Notice that the original treatment (Kolomogorov-Sinai) is suitable to work in an "abstract measure space" (say a standard probability space, one needs minor technicalities), the main problem is to generalize the acting group, for "general amenable groups" (again, with minor technicalities), this has been done by Lindenstrauss in his PhD thesis. $\endgroup$ – Asaf Apr 19 '14 at 17:47
  • 2
    $\begingroup$ @Asaf: "entropy is defined by a dynamical system, not just a space". I disagree with this statement. Arguably, the most basic notion of entropy is Shannon entropy, which assigns to every finite probability space a number measuring its effective size: en.wikipedia.org/wiki/Shannon_entropy $\endgroup$ – Tobias Fritz Apr 21 '14 at 0:46
  • 2
    $\begingroup$ Kolmogorov-Sinai entropy is then a derived quantity defined in terms of the Shannon entropy of a partition, and it measures the scaling of this Shannon entropy with time. As far as I can see, the OP's question is about "entropy" in the sense of Shannon entropy, not Kolmogorov-Sinai entropy. $\endgroup$ – Tobias Fritz Apr 21 '14 at 2:42

This is a theorem of K. Berg, that the Haar measure is the measure of maximal entropy for automorphisms of compact groups. See, for example, these lecture notes.

An information-theoretic approach has been developed in the context of scattering theory, mainly for the unitary group, but I imagine the results are readily transposed to the orthogonal group. The Haar measure maximizes the entropy subject to the constraint that the expectation value of the scattering matrix vanishes, otherwise a more general measure known as the Poisson kernel applies. Here is a review with pointers to the literature.

  • $\begingroup$ Thank you for the references. I will wait for answers about Information Theory and the $O(n)$ case. $\endgroup$ – Cristóbal Guzmán Apr 18 '14 at 21:24
  • $\begingroup$ I think Berg's result is very far from what the OP was asking. $\endgroup$ – Anthony Quas Apr 19 '14 at 2:52
  • $\begingroup$ @AnthonyQuas --- I've added the information-theoretic perspective, hopefully approaching more closely what the OP is looking for. $\endgroup$ – Carlo Beenakker Apr 19 '14 at 5:04
  • $\begingroup$ Thank you, Carlo. Indeed, the setup they use in the paper seems to fit what I am looking for, although I am missing the proofs and derivation of the claims. I haven't got the time to check references where these facts could be proven, so I will wait a couple of days for more answers. $\endgroup$ – Cristóbal Guzmán Apr 20 '14 at 23:24

If you are happy to deal just with measures absolutely continuous with respect to the Haar measure (as you say in one of your comments), then I do not see any problem with the differential entropy (as the reference Haar measure for compact groups has a natural normalization).

Otherwise (i.e., for singular measures) differential entropy does not make much sense. However, in a number of problems it can be quite successfully replaced with the so-called informational dimension and its variants. The model situation is when the limit $$ \lim_{r\to 0} \log \mu B_r(x)/\log r $$ exists and is the same for $\mu$-a.e. point $x$ (here $B_r(x)$ is the $r$-ball centered at $x$). In this case all reasonable definitions of the informational dimension coincide, and this is just the Hausdorff dimension of the measure $\mu$ (i.e., the infimum of the Hausdorff dimensions of sets of full measure).

Both the Hausdorff dimension of a measure and its entropy have the same property: the bigger they are, the more "equidistributed" is the measure. Moreover, in dynamical situations the Hausdorff dimension of a measure and its entropy are quite often proportional.

  • $\begingroup$ Thank you. Is there a hay to measure mutual information in this framework? I imagine that could come simply from $I[X;Y]=H[X]-H[X|Y]$, and since the information dimension works for singular measures there is no problem with the conditional measure. $\endgroup$ – Cristóbal Guzmán Apr 25 '14 at 21:53
  • 2
    $\begingroup$ Yes - one can define the "mutual dimension" as $dim(X)+dim(Y)-dim(X\times Y)$. Considerations like this have been used in the theory of fractals. $\endgroup$ – R W Apr 26 '14 at 9:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.