This is addressed by the classical invariant theory, but the answer is more complicated than for general linear or orthogonal groups (in particular, not all minimal generators are quadratic). Let $k$ be a field of characteristic 0. The group $G=SO_m$ acts on $m\times n$ matrices by the left multiplication and this induces a $G$-action on $A=k[X_{ij}].$ Let us view the variables as the entries of the $m\times n$ generic matrix over $k.$ Then the algebra of invariants $A^G$ is generated by:
1 Scalar products of the columns of the matrix $X.$
2 Order $m$ minors of the matrix $X.$
This is the First Fundamental Theorem (FFT) of classical invariant theory for $SO_m.$ In fact, the elements of the first type generate $O_m$-invariants and the elements of the second type generate $SL_m$-invariants ($SO_m=O_m\cap SL_m$).
Moreover, all relations between these generators are also known (the Second Fundamental Theorem, SFT) and there is a good description of a standard monomial basis of $A^G.$ If I am not mistaken, the last part is due to Laskshmibai and coauthors. A comprehensive modern reference is
Laskshmibai and Raghavan, Standard monomial theory. Invariant theoretic approach. Encyclopaedia of Mathematical Sciences, vol 137 (Invariant Theory and Algebraic Transformation Groups VIII), Springer.