What is special in dimension $2$ (When characterizing isometries using the cofactor matrix)? Let $A$ be a real $n \times n$ matrix. Denote by $\operatorname{cof} A$ The cofactor matrix of $A$. By definition, $A (\operatorname{cof} A)^T=\det A  \cdot I$.
Thus, it is immediate that $A \in \operatorname{SO}_n$ if and only if $$ (**) \operatorname{cof} A =A,\det A =1$$
However, if $n \neq 2$ the condition on the determinant is superfluous:
$ \operatorname{cof} A =A \Rightarrow AA^T=\det A \cdot I \Rightarrow (\det A)^2=(\det A)^n \Rightarrow \det A \in \{1,-1\}$. However, Since $\det A \cdot I=AA^T$ is positive semidefinite $\det A \ge 0$ so $\det A = 1$ and $A \in \operatorname{SO}_n$.
For the case $n=2$, an easy calculation shows $\operatorname{cof} A=A$ if and only if $A$ is a scaled rotation. 
Question:
While the above derivations are easy to do algebraically, I would like to find a more geometric explanation of these results. I think this amounts to obtaining a better geometric interpretation for the cofactor matrix. (I know it measure in some sense the volume of $n-1$ dimensional parallelepiped, see here).
In particular, 
Is there any geometric intution behind the condition $A \in\operatorname{SO}_n \iff (**) \operatorname{cof} A =A,\det A =1$?
Is there any explanation for why dimension $2$ is special?
Note: The condition $(**)$  for characterizing matrices in $\operatorname{SO}_n$ is not a mere game. In some contexts this is the only way to show some transformations are indeed isometries. (For instance in proofs of Reshetnyak’s rigidity theorem).
 A: I don't know how much you want, but the moment you write things in terms of multilinear algebra, everything seems to become pretty transparent.
In general, let $V$ be an $n$-dimensional real vector space and let $L : V \to V$ be a linear transformation. It's well known that the determinant $\det(L)$ of $L$ can be invariantly defined as the unique scalar such that
$$
 \wedge^n L = \det(L) \operatorname{id}_{\wedge^n V}.
$$
What's perhaps a little less well known is that the adjugate $\operatorname{adj}(L)$ of $L$ can be invariantly defined to be the unique linear transformation $V \to V$ such that
$$
 \forall v \in V, \forall \omega \in \wedge^{n-1} V,\quad \operatorname{adj}(L)(v) \wedge \omega = v \wedge (\wedge^{n-1}L)(\omega),
$$
from which you can derive the well-known relationship
$$
 \operatorname{adj}(L) \circ L = \det(L) \operatorname{id}_V
$$
at the level of linear transformations; if $L : \mathbb{R}^n \to \mathbb{R}^n$ is left multiplication by an $n \times n$ matrix $A$, then $\operatorname{adj}(L)$ is precisely left multiplication by the adjugate matrix $\operatorname{adj}(A)$ of $A$. At last, if you give $V$ an inner product, then, once more, $L \in SO(V)$ if and only if $\det(L) = 1$ and $L^\ast = \operatorname{adj}(L)$.
Let's now take a closer look at the case where $L^\ast = \operatorname{adj}(L)$. Since we have $L^\ast L = \det(L) \operatorname{id}_V$, we again find that either $L = 0$ or $L$ is invertible with $\det(L) > 0$. Restricting ourselves to the non-trivial case, if 
$$
 R := \det(L)^{-1/2} L,
$$
then $R \in SO(V)$, so that
$$
 L = (\det(L)^{1/2} \operatorname{id}_V) \circ R
$$
is necessarily a scaled rotation; again, if $n=2$, then, as you've checked, this is the most you can say about $L$, whilst if $n \neq 2$, then you can again conclude that $\det(L) = 1$ and hence that $L = R \in SO(V)$.
A: This is also not a formal answer, but gives another (similar) viewpoint on the difference between dimension 2 and higher dimensions:
When $n=2$, the mapping $A \mapsto \text{cof}\, A$ is a linear, and hence the solutions to $\text{cof}\,A = A$ must form a subspace. In particular, it will be invariant to dilations. In higher dimensions $A \mapsto \text{cof}\, A$ is not linear, and in particular $\text{cof}\,\lambda A = \lambda^{n-1} \text{cof}\,A$.
A: This is somewhat informal. My argument is in the spirit of dimensional analysis in physics.
The essential point is that the cofactor matrix encapsulates $n-1$ dimensional volume, while the dimension of the matrix $A$ in this sense is $1$. (Think the of operator norm, for instance. The "size" of the elements of $A$ roughly corresponds to dilation of length, i.e to one-dimensional volume).
Thus, if $n-1 \neq 1$ it is very unlikely that equality like $A=\operatorname{Cof} A$ will hold, since it compares volumes of different "scales". Indeed, it turns out the only case where this happens is when $A$ is an isometry, i.e there is no difference between different scales\dimensions (think on the toy example $1^d=1$).
Of course, for $n=2$ the incompatibility of scales disappears, so it's reasonable to expect for more instances of $A=\operatorname{Cof} A$, other than the rigid ones. 
Naturally, this heuristic argument does not explain why the scaled rotations are the only ones which stand out, but this is probably related to some specific details regarding exactly how the cofactor encodes what it encodes...
