In answer to II, I have a Penguin book called The Treasury of Mathematics which reprints Cayley's "A memoir on the theory of Matrices". In it he says

"...the term matrix used without qualification is to be understood as meaning a square matrix; in this restricted sense, a set of quantities arranged in the form of a square, e.g.

$$ \begin{array}{c} \begin{pmatrix}a,&b,&c\end{pmatrix} \\\\ \begin{vmatrix}d,&e,&f\\\\g,&h,&k\end{vmatrix} \end{array} $$

..."

So it looks as if that is to be interpreted simply as a 3x3 matrix. He then says that the 3x3 system

$$\begin{eqnarray*} X &= ax+by+cz \\\\ Y & = a'x + b'y + c' z \\\\ Z &= a''x + b'' y + c '' z \end{eqnarray*} $$

"may be more simply represented by"

$$ (X,Y,Z)= \begin{array}{l} \begin{pmatrix}a,&b,&c\end{pmatrix}\\!\\!\\!(x,y,z) \\\\ \begin{vmatrix}a',&b',&c'\\\\a'',&b'',&c''\end{vmatrix} \end{array} $$

Somehow I've broken the formatting on that last one, but it is supposed to look like your final displayed equation without the $(x'\ldots $ bracket.

In answer to II, I have a Penguin book called The Treasury of Mathematics which reprints Cayley's "A memoir on the theory of Matrices". In it he says

"...the term matrix used without qualification is to be understood as meaning a square matrix; in this restricted sense, a set of quantities arranged in the form of a square, e.g.

$$ \begin{array}{c} \begin{pmatrix}a,&b,&c\end{pmatrix} \\\\ \begin{vmatrix}d,&e,&f\\\\g,&h,&k\end{vmatrix} \end{array} $$

..."

So it looks as if that is to be interpreted simply as a 3x3 matrix. He then says that the 3x3 system

$$\begin{eqnarray*} X &= ax+by+cz \\\\ Y & = a'x + b'y + c' z \\\\ Z &= a''x + b'' y + c '' z \end{eqnarray*} $$

"may be more simply represented by"

$$ (X,Y,Z)= \begin{array}{l} \begin{pmatrix}a,&b,&c\end{pmatrix}\!\!\!(x,y,z) \\\\ \begin{vmatrix}a',&b',&c'\\\\a'',&b'',&c''\end{vmatrix} \end{array} $$

Somehow I've broken the formatting on that last one, but it is supposed to look like your final displayed equation without the $(x'\ldots $ bracket.