# What is the “positive part” of the unit ball in $M_n(R)$ ?

In ${\bf M}_n(\mathbb R)$, let us consider the usual operator norm $$\|A\|=\sup\frac{\|Ax\|}{\|x\|},$$ where $\|x\|$ is the Euclidian norm.

The closed unit ball $B$ is the set of contractions (in the terminology used by operator theorists). It is a convex compact subset of ${\bf M}_n(\mathbb R)$. By Krein-Milman (finite dimensional case), it is the convex hull of its subset ${\rm ext}(B)$ of extremal points. It turns out that ${\rm ext}(B)$ is the orthogonal group ${\bf O}_n(\mathbb R)$.

Now, remember that ${\bf O}_n(\mathbb R)$ has two connected components, a positive one ${\bf SO}_n(\mathbb R)$ and a negative one ${\bf O}_n^-(\mathbb R)$.

What is the convex hull of ${\bf SO}_n(\mathbb R)$ ?

Clearly, it is a compact convex subset, included in $B$. It is a strict subset of $B$, because it does not meet ${\bf O}_n^-(\mathbb R)$. The title refers to the "positive part" of $B$, but this could be inappropriate, in the sense that it could meet the convex hull of ${\bf O}_n^-(\mathbb R)$ non-trivially.

Remark also that this convex hull is invariant under multiplication at right or left by an element of ${\bf SO}_n(\mathbb R)$. Therefore it would be enough to decide which diagonal matrices ${\rm diag}(a_1,\ldots,a_n)$ with $|a_1|\le a_2\le\cdots\le a_n$ it contains.

When $n=2$, ${\bf SO}_n(\mathbb R)$ is a circle and its convex hull is a disk, obviously a much smaller set (even from the dimensional point of view) than $B$.

-
If $B$ is open, then $B$ has no extremal points, and so is not the convex hull of the set of its extremal points. $\hspace{.2 in}$ If $B$ is closed, then $B$ has members that are not contractions. $\;\;$ – Ricky Demer Feb 29 '12 at 8:44
Ricky, it is customary in many places/books/texts/whatever in analysis to use the word "contraction" to mean "distance non-increasing". This is how almost all practising operator theorists and most functional analysts I've met use the word, for instance – Yemon Choi Feb 29 '12 at 8:44
The answer for $n=3$ is given in $\S 4.1$ of arxiv.org/abs/0911.5436. In $\S 4.4$ of ibid, there's a discussion of some properties of the convex hull of $SO(n)$ for larger $n$. – Laurent Berger Feb 29 '12 at 15:31
Merci, Laurent ! – Denis Serre Feb 29 '12 at 15:47