inverse-closed matrix spaces

Question

Is there a known characterization of such spaces?

An example: the space of $n \times n$ matrices spanned by $I$ and $J$ (the identity and all-ones matrices, respectively) is inverse closed by the Sherman-Morrison formula.

A possible question of interest would be the maximum dimension of a non-trivial inverse-closed space.

I am reminded here of the work on spaces of matrices of bounded rank about which not a little is known but maybe it's a very false analogy.

EDIT: After reading Denis Serre's neat answer I started thinking what happens in the singular case. The Moore-Penrose inverse of $A$ in general is not polynomial in $A$. It does not even always commute with $A$. But in a paper by Edward Wong, Does the Generalized Inverse of A Commute with A? (Mathematics Magazine, Vol. 59, No. 4 (Oct., 1986), pp. 230-232) it was shown that $A^{\dagger}$ is a polynomial in $A$ if and only if $A$ and $adj(A)$ have the same row-reduced echelon form, so that's some answer to the generalized question.

Answer 1

Such matrix subspaces have been characterized: They are Jordan algebras. A bit more generally, linear structure is preserved in inversion if a matrix subspaces is equivalent to a Jordan algebra.

(Jordan algebra is closed under the product MN+NM.) Equivalence means that you are allowed to multiply with an invertible matrix from the left and with an invertible matrix from the right.

Reference: (Marko Huhtanen, Differential geometry of matrix inversion. Math. Scand., 107, pp. 267-284, 2010.)

Answer 2

Take any matrix $n\times n$ $A$, then $E$ the sub-algebra spanned by $A$ ,that is the set of $P(A)$ with $P$ polynomials. This space is inverse-closed (Caylet-Hamilton). Its dimension may be $n$.

Answer 3

Matrices for which a given vector is an eigenvector are inverse closed. This subspace has dimension $n^2-n+1$.

Answer 4

The upper triangular $n \times n$ matrices are inverse-closed, and this subspace has dimension $n(n+1)/2$.