12
$\begingroup$

Let $G$ be a locally compact group. A measure $\mu$ is the right-Haar measure on $G$ if for every $g\in G$ and $E\subseteq G$ Borel set $\mu(Eg)=\mu(E)$. It is known that every locally compact group has a unique right-Haar measure (up to a multiplicative constant). Similarly one can define the left-Haar measure as being invariant for left shifts. However, in the case $G$ is compact this distinction is irrelevant, since both left and right Haar measures are equivalent up to a constant (note that $G$ is not nececssarily commutative, so being left and right Haar measures are in fact two different properties).

I am interested on a natural notion of entropy for measures on compact groups. In this case I expect the Haar measure to maximize the entropy. Is there an analog of information theory developed in this framework? I am particularly interested in the case where

I am particularly interested on the case where $G=O(n)$, the group of orthogonal transformations of $\mathbb{R}^n$.

Any references or insights are greatly welcome!

---------------------------- Update ------------------------------

Due to a request, I will further explain. Before I start, for notation and definitions I refer to Cover & Thomas book (Elements of Information Theory).

My question is very much related to Intrinsic significance of differential entropy: in the case of real random variables, differential entropy is troublesome due to the fact that there is no canonical reference space (in the discrete case, we are always working with r.v.'s over $\{1,\ldots,n\}$), so for example differential entropy is not invariant under re-scalings, since $$ h(a X) = h(X) + \log(a), $$ as opposed to the discrete case where $h(aX)=h(X)$.

However, in the case your r.v.'s are defined over a compact group $G$ (we may restrict here to the ones that are absolutely continuous w.r.t. Haar), the re-scaling problem disappears, as we have a reference space $G$. I wonder if having this reference space helps at all in terms of having information-theoretic identities/inequalities similar to the discrete case (e.g., Data Processing Inequality).

Finally, from what I have been reading lately, it seems that this is hopeless: just the fact of having an infinite measure space seems to break down all nice properties of the discrete entropy. However, I want to keep this question open to further comments from people with more acquaintance with information theory.

$\endgroup$
11
  • 1
    $\begingroup$ In a compact grouple the distinction between left invariant and right invariant is not meaningless but irrelevant or something —compactness does not make the distinction lose its meaning! $\endgroup$ Apr 18, 2014 at 19:26
  • 1
    $\begingroup$ Absolutely! I will rephrase it. Thank you $\endgroup$ Apr 18, 2014 at 21:04
  • 1
    $\begingroup$ entropy is defined by a dynamical system, not just a space. Which dynamical system are you considering here? Notice that the original treatment (Kolomogorov-Sinai) is suitable to work in an "abstract measure space" (say a standard probability space, one needs minor technicalities), the main problem is to generalize the acting group, for "general amenable groups" (again, with minor technicalities), this has been done by Lindenstrauss in his PhD thesis. $\endgroup$
    – Asaf
    Apr 19, 2014 at 17:47
  • 2
    $\begingroup$ @Asaf: "entropy is defined by a dynamical system, not just a space". I disagree with this statement. Arguably, the most basic notion of entropy is Shannon entropy, which assigns to every finite probability space a number measuring its effective size: en.wikipedia.org/wiki/Shannon_entropy $\endgroup$ Apr 21, 2014 at 0:46
  • 2
    $\begingroup$ Kolmogorov-Sinai entropy is then a derived quantity defined in terms of the Shannon entropy of a partition, and it measures the scaling of this Shannon entropy with time. As far as I can see, the OP's question is about "entropy" in the sense of Shannon entropy, not Kolmogorov-Sinai entropy. $\endgroup$ Apr 21, 2014 at 2:42

2 Answers 2

11
$\begingroup$

This is a theorem of K. Berg, that the Haar measure is the measure of maximal entropy for automorphisms of compact groups. See, for example, these lecture notes. [broken link]

An information-theoretic approach has been developed in the context of scattering theory, mainly for the unitary group, but I imagine the results are readily transposed to the orthogonal group. The Haar measure maximizes the entropy subject to the constraint that the expectation value of the scattering matrix vanishes, otherwise a more general measure known as the Poisson kernel applies. Here is a review with pointers to the literature.

$\endgroup$
6
  • $\begingroup$ Thank you for the references. I will wait for answers about Information Theory and the $O(n)$ case. $\endgroup$ Apr 18, 2014 at 21:24
  • $\begingroup$ I think Berg's result is very far from what the OP was asking. $\endgroup$ Apr 19, 2014 at 2:52
  • $\begingroup$ @AnthonyQuas --- I've added the information-theoretic perspective, hopefully approaching more closely what the OP is looking for. $\endgroup$ Apr 19, 2014 at 5:04
  • $\begingroup$ Thank you, Carlo. Indeed, the setup they use in the paper seems to fit what I am looking for, although I am missing the proofs and derivation of the claims. I haven't got the time to check references where these facts could be proven, so I will wait a couple of days for more answers. $\endgroup$ Apr 20, 2014 at 23:24
  • $\begingroup$ Two of the links no longer work, would you be able to provide new ones? Thanks. $\endgroup$ Jan 16 at 17:20
5
$\begingroup$

If you are happy to deal just with measures absolutely continuous with respect to the Haar measure (as you say in one of your comments), then I do not see any problem with the differential entropy (as the reference Haar measure for compact groups has a natural normalization).

Otherwise (i.e., for singular measures) differential entropy does not make much sense. However, in a number of problems it can be quite successfully replaced with the so-called informational dimension and its variants. The model situation is when the limit $$ \lim_{r\to 0} \log \mu B_r(x)/\log r $$ exists and is the same for $\mu$-a.e. point $x$ (here $B_r(x)$ is the $r$-ball centered at $x$). In this case all reasonable definitions of the informational dimension coincide, and this is just the Hausdorff dimension of the measure $\mu$ (i.e., the infimum of the Hausdorff dimensions of sets of full measure).

Both the Hausdorff dimension of a measure and its entropy have the same property: the bigger they are, the more "equidistributed" is the measure. Moreover, in dynamical situations the Hausdorff dimension of a measure and its entropy are quite often proportional.

$\endgroup$
2
  • $\begingroup$ Thank you. Is there a hay to measure mutual information in this framework? I imagine that could come simply from $I[X;Y]=H[X]-H[X|Y]$, and since the information dimension works for singular measures there is no problem with the conditional measure. $\endgroup$ Apr 25, 2014 at 21:53
  • 2
    $\begingroup$ Yes - one can define the "mutual dimension" as $dim(X)+dim(Y)-dim(X\times Y)$. Considerations like this have been used in the theory of fractals. $\endgroup$
    – R W
    Apr 26, 2014 at 9:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.