Is SO(4) a subgroup of SU(3)? $\DeclareMathOperator\SO{SO}\DeclareMathOperator\SU{SU}$I want to write a $3 \times 3$ complex-matrix representation of $\SO(4)$, for example, we know that $\SO(5)$ is a subgroup of $\SU(4)$, so we can write a $4 \times 4$ complex-matrix representation of $\SO(5)$. There is a paper which gives this representation: Mapping two-qubit operators onto projective geometries. This paper mentions which matrices form the $\SO(5)$ group, and Manipulating two-spin coherences and qubit pairs contains the matrices explicitly (both papers are by A.R.P. Rau).
I want to know if $\SO(4)$ is also a subgroup of $\SU(3)$, and if so, can we find a $3 \times 3$ complex-matrix representation of $\SO(4)$?
 A: No. There is probably a straightforward representation-theoretic argument, but I am too ignorant of the subject to give one, so here is a topological argument.
If $H \subset G$ are Lie groups with $H$ closed in $G$, then $G/H$ has the natural structure of a smooth manifold without boundary. If $G$ is compact, so is $G/H$, as it is the continuous image of a compact space; if $H$ is compact the closed-ness condition is automatic.
Now $\dim SU_3 = 8$ and $\dim SO_4 = 6$. Therefore, if there existed some group embedding $j: SO_4 \hookrightarrow SU_3$, the quotient $X = SU_3/j(SO_4)$ must be a compact 2-manifold without boundary.
Furthermore, the fibering $SO_4 \xrightarrow{j} SU_3 \to X$ induces the long exact sequence of homotopy groups $$\pi_2 SU_3 \to \pi_2 X \to \pi_1 SO_4 \to \pi_1 SU_3 \to \pi_1 X \to \pi_0 SO_4.$$ Filling in the values we know for these groups, we find that there is a long exact sequence of groups $$0 \to \pi_2 X \to \Bbb Z/2 \to 0 \to \pi_1 X \to 0.$$ Therefore $\pi_2 X \cong \Bbb Z/2$ and $\pi_1 X \cong 0$. This is a contradiction; the only simply connected compact surface without boundary is the 2-sphere, which has $\pi_2 X = \Bbb Z$.
A: Maybe the simplest argument, if you know something about compact Lie groups, is that SO(4) and SU(3) both have rank 2, i.e., they each contain a maximal torus, which is $S^1\times S^1$.  Since all maximal tori are conjugate in a compact Lie group, if SO(4) were a subgroup, then, after a conjugation in SU(3), you can assume that SO(4) contains the diagonal subgroup of SU(3) (which is a maximal torus).
But then the Lie algebra of SO(4) (which has dimension 6) would have to be the Lie algebra of the maximal torus plus the sum of two of the three (two-dimensional) weight spaces of the maximal torus acting on the Lie algebra of SU(3).  However, the Lie bracket of any two of these weight spaces is the third weight space, so there can't be such a 6-dimensional Lie subalgebra of su(3).
A: No. Indeed $\mathrm{SO}(4)$ satisfies the following condition (which is a first-order existential formula) but not $\mathrm{SU}(3)$:
$$\exists w,x,y,z: [x,w]\neq 1\neq [y,z],\; [w,y]=[w,z]=[x,y]=[x,z]$$
(it just says there are two commuting pairs of non-abelian subgroups).
Indeed $\mathrm{SO}(4)$ even contains a copy of the direct product of two non-abelian free groups.
Let's show this doesn't exist in $\mathrm{U}(3)$. Suppose by contradiction we have such a 4-tuple. Since the $\mathbf{C}$-subalgebra generated by $w,x$ is semisimple and non-commutative, after conjugation it is either $\mathrm{M}_3(\mathbf{C})$ or the block-diagonal $\mathrm{M}_2(\mathbf{C})\times \mathrm{M}_1(\mathbf{C})$. So its centralizer is commutative in both cases, contradiction.
In particular there is no injective homomorphism $\mathrm{SO}(4)\to\mathrm{U}(3)$ (no continuity required).

Here's an alternative argument to show that the centralizer of every non-solvable subgroup $G$ in $\mathrm{GL}_3(C)$, with $C$ any field, is abelian. We can suppose that $C$ is algebraically closed. Indeed, choose a triangulation by blocks of $G$ with irreducible diagonal blocks. If it's trivial (i.e., $G$ is irreducible), the centralizer is reduced to scalars. So it's $1+2$ or $2+1$ (if it were $1+1+1$, $G$ would be solvable). In both case, the subalgebra generated by $G$ contains a diagonal copy of $\mathrm{M}_2(C)\oplus \mathrm{M}_1(C)$. So its centralizer is commutative.
Since $\mathrm{SO}(4)$ contains two pairwise copies of the binary icosahedral group of order $120$, it also shows that some finite subgroup of $\mathrm{SO}(4)$ does not embed into $\mathrm{SU}(3)$ (and even in $\mathrm{GL}_3$).

Also, as pointed out by Friedrich Knop in a comment, the premise of the question is slightly false: $\mathrm{SO}(5)$ actually has no injective homomorphism into $\mathrm{SU}(4)$, regardless of any continuity assumption. Indeed, $\mathrm{SO}(5)$ has a 2-elementary subgroup of order $2^4$, while $\mathrm{SU}(4)$ has none. (One indeed has a continuous homomorphism $\mathrm{Spin}(5)\to\mathrm{SU}(4)$, or $\mathrm{SO}(5)\to\mathrm{PSU}(4)$.) So in this case the question is sensitive to coverings.
In contrast, no connected Lie group locally isomorphic to $\mathrm{SO}(4)$ (these are $\mathrm{SU}(2)^2$, $\mathrm{SO}(3)^2$, $\mathrm{SO}(3)\times\mathrm{SU}(2)$, and $\mathrm{SO}(4)$) can be embedded abstractly into a connected Lie group locally isomorphic to $\mathrm{SU}(3)$ (these are $\mathrm{SU}(3)$ and $\mathrm{PSU}(3)$). In each case some finite subgroup (which can be chosen as central quotient of the direct square of the binary icosahedral group of order $120$) testifies the obstruction.
A: It isn't too bad to describe the irreducible representations of $SO(4)$. We can realize $SO(4)$ as $SU(2) \times SU(2) / \langle (- \text{Id}, - \text{Id})\rangle$. To see this, identify $\mathbb{R}^4$ with the quaternions, and identify $SU(2)$ with the norm $1$ quaternions. For $(u,v) \in SU(2) \times SU(2)$, the map $q \mapsto uqv^{-1}$ from the quaternions to themselves preserves the norm, so we get a map $SU(2) \times SU(2) \to SO(4)$; one can check that this map is surjective and has kernel the two element group generated by $(- \text{Id}, - \text{Id})$.
The irreducible representations of $SU(2)$ are well known to be $\text{Sym}^m(\mathbb{C}^2)$ for $m \geq 0$. This has dimension $m+1$, and $- \mathrm{Id}$ acts by $(-1)^m$. So the irreps of $SU(2) \times SU(2)$ are $\text{Sym}^m(\mathbb{C}^2) \boxtimes \text{Sym}^n(\mathbb{C}^2)$, with dimension $(m+1)(n+1)$, and with $(- \text{Id}, - \text{Id})$ acting by $(-1)^{m+n}$. This representation factors through $SO(4)$ if and only if $m+n$ is even, so the irreps of $SO(4)$ are $\text{Sym}^m(\mathbb{C}^2) \boxtimes \text{Sym}^n(\mathbb{C}^2)$ for $m+n \equiv 0 \bmod 2$. Whenever $m \equiv n \equiv 0 \bmod 2$, then the kernel contains the subgroup $(\pm\text{Id}, \pm\text{Id})$ with 4 elements, so the smallest faithful representation is $\mathbb{C}^2 \boxtimes \mathbb{C}^2$ (so $m=n=1$), with dimension $4$.
A: Here is a very minimalistic argument.
$SO(4)$ contains a rank $3$ elementary abelian $2$-group (product of three groups of order two), namely the group of diagonal matrices in $SO(4)$.
$SU(3)$ does not. Any abelian subgroup of $SU(3)$ is conjugate to a group of diagonal matrices, and among the diagonal matrices there are only four whose squares are the identity
A: In addition to all the completely correct answers to this question, it's probably worth spelling out some of the misconceptions that are common to physicists who work in this area and giving the physics answer.
The main thing that arises all the time is that physicists often conflate representations of a group and of its Lie algebra. The reason one can get away with this is that really in physics one deals with projective representations which are the same (in most common cases) as representations of the universal cover of the group. Thus, distinctions like the difference between $Spin(4) \simeq SU(2) \times SU(2)$ and $SO(4) \simeq \left(SU(2) \times SU(2)\right) / \mathbb{Z}_2$ are lost.
The other slightly odd (and not so common) misconception that seems to be here is that you need a group to be a subgroup of $U(n)$ to have a complex $n$ dimensional representation. A map into $U(n)$ is more than enough for obvious reasons.
Taking these into account, the original statement in the question becomes that you have a map from $Spin(5) \simeq Sp(4) \to SU(4)$ giving a four dimensional representation. And the answer to the physics version of the question is "yes-ish", because there is a map from $Spin(4) \to SU(2)$ given by projection onto one of the two $SU(2)$ components (say) and then you can take the $\mathbb{Z}_2$ quotient to get a map from $SO(4) \to SO(3)$.
Or, to put it another way, there is a perfectly cromulent, but not faithful, three dimensional representation of $SO(4)$.
