Emergence of the orthogonal group

Do we know what mathematician first considered, and perhaps named, what we call the group $$\mathrm O(n)$$, or $$\mathrm{SO}(n)$$, for some $$n>3$$?

I mean it specifically as group (not Lie algebra) acting on Euclidean $$n$$-space. For $$n=3$$ Jordan (1868) seems a definite upper bound, but for higher $$n$$ it seems not clear to me that even Cartan (1894) thought in those terms, describing as he does $$\mathsf B_l$$ and $$\mathsf D_l$$ as “projective groups of a nondegenerate surface of second order in spaces of $$2l$$ and $$2l-1$$ dimensions.” Also please disregard any implicit occurrence of $$\mathrm{SO}(4)$$ in quaternion theory.

• In Jordan's 1868 paper all groups are finite, so how could the real group $\mathrm{(S)O}(3)$ be defined? – YCor Jun 18 at 6:12
• @YCor §§14-15 “Il nous reste à examiner le cas où parmi les rotations du groupe, il en existe une $A_\varrho$ dont l’amplitude $\varrho$ soit infiniment petite (...) Le groupe contenant une rotation quelconque autour d’un quelconque de ces axes, contiendra toutes les rotations possibles (...) 6ème type. Il est formé par l’ensemble de tous les mouvements de rotation possibles.” – Francois Ziegler Jun 18 at 11:51
• Oh, thanks. My knowledge on that part of history was mainly based on Wüssing's book, and I actually see that this 1868 memoir is indeed mentioned therein (p196-197), which I had overseen or forgotten. – YCor Jun 18 at 12:28

Your quote about Cartan thinking of $$B_n$$ and $$D_n$$ as 'projective groups..." is actually Cartan describing the lowest dimensional homogeneous space of these groups (except, of course, for a few exceptional cases such as $$D_2$$, which is not simple, and therefore should be left out of the description).

If you go just a little bit further in Cartan's 1894 Thesis, to Chapitre VIII, Section 9, you'll see that Cartan describes linear representations as well. For example, of $$B_\ell$$, he writes "C'est le plus grand groupe linéare et homogéne de l'espace à $$2\ell{+}1$$ dimensions qui laisse invariante la forme quadratique $${x_0}^2 + 2x_1x_{1'} +2x_2x_{2'} + \cdots + 2x_\ell x_{\ell'}"$$ with a similar description for $$D_\ell$$.

In fact, he gives the lowest dimensional representation of each of the simple groups over $$\mathbb{C}$$, including the exceptional ones and, except for $$\mathrm{E}_8$$, he explicitly describes the equations that define the representation. For example, he writes down an explicit homogeneous cubic in 27 variables and states that $$\mathrm{E}_6$$ is the the subgroup of $$\mathrm{GL}(27,\mathbb{C})$$ that preserves this cubic form.

For the summary theorem on the linear representations, see Chapitre VIII, Section 10, where he lists each of the lowest representations and notes the various low dimensional exceptional isomorphisms as well.

Remark 1: Cartan continues to refer to groups of type $$B$$ and $$D$$ merely as "the largest groups preserving a quadratic form in $$n$$ variables" or similar terms for a long time. Even in his papers of 1913–1915 classifying the real forms of the complex simple Lie groups, he uses such terminology, though he clearly finds the special case of the compact real forms of special interest.

The first place that Cartan actually refers to 'orthogonal groups' that I can recall are in his 1926–27 papers on the classification of Riemannian symmetric spaces. There, he begins referring to any subgroup of $$\mathrm{GL}(n,\mathbb{R})$$ that preserves a quadratic form as 'an orthogonal group' and then, later, finally refers to the maximal group that preserves a positive definite quadratic form as 'the orthogonal group'. I don't recall when or whether he used any notation such as $$\mathrm{O}(n)$$ or $$\mathrm{SO}(n)$$.

Whether the term 'orthogonal group' was original to him, I can't say. By that time, of course, Weyl had already started his research on compact Lie groups, and it may be that Weyl had already used the term 'orthogonal group' well before Cartan.

Remark 2: Euler's article (Problema algebraicum ob affectiones prorus singulares memorabile, Novi commentarii academiae scientiarum Petropolitanae 15 (1770) 1771, 75–106) discusses the problem of parametrizing the solutions of the equation $$A^TA = I_n$$ where $$A$$ is an $$n$$-by-$$n$$ matrix for $$n=3$$, $$4$$, and $$5$$, particularly the rational solutions. He does not use the terminology 'orthogonal' or 'group'. Nevertheless his article does contain some remarkable formulae that clearly anticipate the development of the algebra of quaternions.

For example, identifying $$\mathbb{R}^4$$ with the quaternions $$\mathbb{H}$$ in the usual way, it is a now-standard fact that every special orthogonal linear transformation $$M$$ of $$\mathbb{R}^4=\mathbb{H}$$ can be written, using quaternion multiplication, in the form $$M(X) = A\,X\,\bar B$$ where $$A$$ and $$B$$ are unit quaternions and $$X\in\mathbb{H}$$. (This is now the usual way that the double cover $$\mathrm{Spin}(3)\times\mathrm{Spin}(3)\to\mathrm{SO}(4)$$ is introduced.) Meanwhile conjugation $$c:\mathbb{H}\to\mathbb{H}$$ is orthogonal but has determinant $$-1$$, so every element of the non-identity component of $$\mathrm{O}(4)$$ can be written as $$M'(X) = Ac(X)\bar B = A\,\bar X\, \bar B = A\overline{BX} = Ac(BX).$$ Remarkably, Euler gives this formula for parametrizing $$\mathrm{O}(4)$$ in the form of the product of matrices $$L_A\,c\,L_B$$ (where $$L_P$$ denotes left multiplication by the quaternion $$P$$), many years before the 'official' discovery of quaternions.

• Thanks! I should definitely have read further. Would you say that Cartan has a chance of being first to “consider” the group, though not name it other than by one subscripted letter? Ironically Frobenius (1878) seems to do almost the opposite... – Francois Ziegler Jun 18 at 12:21
• @FrancoisZiegler: I wouldn't venture to say that. From what Carlo wrote in his answer, it would appear that Hurwitz already was dealing with $\mathrm{SO}(n)$ and $\mathrm{U}(n)$ as matrix groups in 1897, which seems pretty advanced for the time. Are you sure that Lie and/or Killing didn't recognize that the stabilizer groups of nondegenerate quadratic forms were of Lie's type B or D? Since the relation between 'linear fractional transformations' and $\mathrm{SL}(2)$ was well understood by Lie, you would expect that he'd have realized that 'projective groups' were matrix groups in disguise. – Robert Bryant Jun 18 at 13:24
• I don’t know! The problem with Lie and Killing is that they are hard to catch talking about anything but Lie algebras. E.g. Lie (1893, p. 317) describing $\mathfrak{so}(n)$ as “the group $x_\nu p_k-x_k p_\nu\quad (\nu,k=1\dots n)$”. – Francois Ziegler Jun 18 at 14:11
• @FrancoisZiegler: Hmmm. I remember that Lie used $p_k$ for what we call the vector field $\frac{\partial}{\partial x_k}$, so he's discussing the group acting on $\mathbb{R}^n$ with 'infinitesimal generators' $x_\nu\,\frac{\partial}{\partial x_k} -x_k\,\frac{\partial}{\partial x_\nu}$, which is, of course, what we now call the (special) orthogonal group acting on $\mathbb{R}^n$. He surely knew that this was a linear action on $\mathbb{R}^n$. Lie's whole point was that the infinitesimal generators determine the group, so specifying the group by its Lie algebra would have been natural for him. – Robert Bryant Jun 18 at 14:49
• @FrancoisZiegler: Thanks for pointing out Euler's 1771 paper. I had a look at it. Doesn't he consider $\mathrm{O}(5)$ as well, starting in $\S28$? (Of couse, he didn't name it as a group.) Also, he has the formula for the double cover $S^3\to\mathrm{SO}(3)$ in $\S33$ and quaternion multiplication in $\S34$! I had not known this before. – Robert Bryant Jun 19 at 11:02

There may be an earlier source, but Adolf Hurwitz 1897 is one upper bound:

A. Hurwitz, Über die Erzeugung der Invarianten durch Integration, Nachr. Ges. Wiss. Göttingen (1897), 71–90.

Hurwitz’s paper introduced and developed the notion of an invariant measure for the matrix groups SO(N) and U(N). He also specified a calculus from which the explicit form of these measures could be computed in terms of an appropriate parametrisation — Hurwitz chose to use Euler angles. This enabled him to define and compute invariant group integrals over SO(N) and U(N).

• Thanks. Indeed Hurwitz definitely speaks of “der orthogonalen Gruppe” (pp. 72, 75). He also cites e.g. Kronecker (1890) who had “die Gesammtheit der orthogonalen Transformationen” (p. 376), “die Mannigfaltigkeit der orthogonalen Transformationen” (p. 455). – Francois Ziegler Jun 18 at 13:56
• Perhaps the first to speak of orthogonal transformations (though not orthogonal group) is J. J. Sylvester in A Demonstration of the Theorem that every Homogeneous Quadratic Polynomial is reducible by real orthogonal substitutions to the form of a sum of Positive and Negative Squares, Phil. Mag. (4) 4 (1852) 138-142. He refers to Boole who earlier called them rectangular transformations in On the Theory of Linear Transformations (1851, p. 88). – Francois Ziegler Jun 22 at 15:59
• I guess the switch may have been because “orthogonal matrix” was preferable to “rectangular matrix”. As to groupe orthogonal, the name seems to have been first used by Jordan in Traité des substitutions et des équations algébriques (1870), p. 155. – Francois Ziegler Jun 22 at 19:58