show/hide this revision's text 2 added 2 characters in body

I think you mean an $N\times N$ matrix in the first sentence.

Then the answer is the following: Then if you are interested in the most general case the answer is that there is no such normal form (if you are interested in one valid for all $N$). You can translate this problem by asking for representations of the algebra $k\langle X,Y\rangle$, the polynomial ring in two non-commuting variables. The representation theory is undecidable. Look also for the definition of a wild algebra.

If your algebra has special properties, then there might be an answer. For example, if you ask for pairs of commuting nilpotent 2-nilpotent matrices, you can transfer this to the problem of the representation theory of $k[x,y]/(x^2,y^2)$, which is well-known, look for special biserial algebras.

With three variables, it does not even matter, what kind of additional conditions you impose (except maybe for trivial ones like two matrices to be equal or restriction of dimension). You will never be able to find a normal form (this is also related to wild algebras, see the $3$-Kronecker quiver)

If you need additional references, please ask in the comment.
show/hide this revision's text 1

I think you mean an $N\times N$ matrix in the first sentence.

Then the answer is the following: Then if you are interested in the most general case the answer is that there is no such normal form (if you are interested in one valid for all $N$). You can translate this problem by asking for representations of the algebra $k\langle X,Y\rangle$, the polynomial ring in two non-commuting variables. The representation theory is undecidable. Look also for the definition of a wild algebra.

If your algebra has special properties, then there might be an answer. For example, if you ask for pairs of commuting nilpotent matrices, you can transfer this to the problem of the representation theory of $k[x,y]/(x^2,y^2)$, which is well-known, look for special biserial algebras.

With three variables, it does not even matter, what kind of additional conditions you impose (except maybe for trivial ones like two matrices to be equal or restriction of dimension). You will never be able to find a normal form (this is also related to wild algebras, see the $3$-Kronecker quiver)

If you need additional references, please ask in the comment.