While trying to use Minkowski's theorem to calculate the (left) class number of a noncommutative ring, I ran into the following problem:

What is the volume of the largest symmetric convex subset $S$ of $\mbox{Mat}_3(\mathbb{R})$ such that every matrix in this subset has determinant at most $1$? In particular, is there such a set with volume greater than 885?

The best subset that I have been able to find so far is

$S_1 = \{M\in\mbox{Mat}_3(\mathbb{R})\mid \forall O\in O(3,\mathbb{R}), \mbox{ Tr}(MO) \le 3\}$.

If we apply Gram-Schmidt to the rows of $M$ and use the resulting orthonormal basis for the columns of an orthogonal matrix $O$, then by the AM-GM inequality we can conclude that $\det M \le 1$ from $\mbox{Tr}(MO)\le 3$. Furthermore, if $S$ is any symmetric convex set of matrices of determinant at most $1$ which contains an orthogonal matrix $O$, then every $M\in S$ must satisfy $\mbox{Tr}(MO^{-1}) \le 3$ by considering the tangent hyperplane to the collection of matrices with determinant $1$. Thus $S_1$ is the largest such set which contains all orthogonal matrices.

Unfortunately, it's incredibly difficult to calculate the volume of $S_1$. All I've been able to do so far is to note that it contains the set $S_r$ of matrices such that the sum of the norms of the rows is at most $3$, and that the volume of $S_r$ is $\frac{972\pi^3}{35}\approx 861.089$. If we let $S_c$ be the similar set obtained by considering the columns, then Monte-Carlo integration indicates that the volume of $S_r\cup S_c$ is around $1050$, but I haven't been able to prove this.

What is the volume of $S_1$? If this is too hard, what is the volume of $S_r\cup S_c$?

## Motivation

The ring I am dealing with is $\mathbb{Z}\langle a,x\rangle/(a^3 = a-1, x^3 = 3x-1, xa = a(x^2-2)+2-x)$. It is an order in the central simple algebra over $\mathbb{Q}$ with invariants $1/3$ at $3$, $2/3$ at $2$, and $0$ at every other prime. The discriminant of this ring is $-6^6$ (as one can verify by working out the trace form by hand).

Since the associated central simple algebra has nontrivial invariants at $2$ and $3$, and since $\mbox{Nm}(x+1) = -3$ and $(a+xa-1)^3 = 2$, all left ideals of norm at most $4$ are principal. If we tensor this ring up to $\mathbb{R}$ it becomes $\mbox{Mat}_3(\mathbb{R})$, so we can think of it as a lattice in $\mbox{Mat}_3(\mathbb{R})$ of covolume $\sqrt{6^6}$. By Minkowski's bound, every left ideal class contains a representative of norm at most $\left(\frac{2^9\sqrt{6^6}}{\mbox{Vol}(S)}\right)^{1/3}$, so if we can find a set $S$ with volume at least $885$ then we can conclude that this ring has left class number $1$.