Is there a known characterization of such spaces?

An example: the space of $n \times n$ matrices spanned by $I$ and $J$ (the identity and all-ones matrices, respectively) is inverse closed by the Sherman-Morrison formula.

A possible question of interest would be the maximum dimension of a non-trivial inverse-closed space.

I am reminded here of the work on spaces of matrices of bounded rank about which not a little is known but maybe it's a very false analogy.

EDIT: After reading Denis Serre's neat answer I started thinking what happens in the singular case. The Moore-Penrose inverse of $A$ in general is **not** polynomial in $A$. It does not even always commute with $A$. But in a paper by Edward Wong, *Does the Generalized Inverse of A Commute with A?* (Mathematics Magazine, Vol. 59, No. 4 (Oct., 1986), pp. 230-232) it was shown that $A^{\dagger}$ is a polynomial in $A$ if and only if $A$ and $adj(A)$ have the same row-reduced echelon form, so that's some answer to the generalized question.