Question

We learned in linear algebra that an $N \times N$ matrices can be placed into jordan normal forms with "blocks": $$ \left[ \begin{array}{cccc} \lambda & 1 & 0 & 0 \\ 0 & \lambda & 1 & 0 \\ 0 & 0 & \lambda & 1 \\ 0 & 0 & 0 & \lambda \end{array} \right] \hspace{0.25in}\text{or }\hspace{0.25in} \left[ \begin{array}{cccc} \lambda & 0 & 0 & 0 \\ 0 & \lambda & 0 & 0 \\ 0 & 0 & \lambda & 0 \\ 0 & 0 & 0 & \lambda \end{array} \right]$$

What is the analogue of "blocks" for a pair of $N \times N$ matrices, $\phi,\psi:V \to V$ ? and for a triple of matrices. $\phi, \psi,\rho: V \to V $ ?

Answer 1

I think you mean an $N\times N$ matrix in the first sentence.

Then the answer is the following: Then if you are interested in the most general case the answer is that there is no such normal form (if you are interested in one valid for all $N$). You can translate this problem by asking for representations of the algebra $k\langle X,Y\rangle$, the polynomial ring in two non-commuting variables. The representation theory is undecidable. Look also for the definition of a wild algebra.

If your algebra has special properties, then there might be an answer. For example, if you ask for pairs of commuting 2-nilpotent matrices, you can transfer this to the problem of the representation theory of $k[x,y]/(x^2,y^2)$, which is well-known, look for special biserial algebras.

With three variables, it does not even matter, what kind of additional conditions you impose (except maybe for trivial ones like two matrices to be equal or restriction of dimension). You will never be able to find a normal form (this is also related to wild algebras, see the $3$-Kronecker quiver)

If you need additional references, please ask in the comment.

Answer 2

For pairs of matrices $(A,B)$ (or rather, we prefer to think about the pencil $Ax-B$), there is the Weierstraß canonical form: there are $P,Q$ such that $PAQ$ and $PBQ$ are direct sums of blocks of four easy-to-describe types. I think this is described in the book by Gantmacher. There surely is a short description in Handbook of linear algebra by Hogben.

For triples of matrices under the same equivalence class, there should be no canonical form.