The general criterion is:
Given connected compact Lie groups $G,H$, $\mathrm{Hom}(G,H)$ has finitely many components if and only if $H$ is semisimple or $G=1$.
Proof.
If $G=1$ there is nothing to prove.
Assume that H is not semisimple and $G\neq 1$. Then there exists a 1-dimensional torus $T$ in $H$ and a closed normal subgroup $N$ of codimension 1 such that $TN=H$. Define $T=T\cap N$, so that $Z$ is finite and define $T'=T/Z$. Then $\mathrm{Hom}(T',G)$ can be identified to the set of elements in $\mathfrak{g}$ whose exponential is 1; we have already seen that it has infinitely many components.
We have a continuous restriction map $\mathrm{Hom}(H,G)\to \mathrm{Hom}(T,G)$. Its image contains $\mathrm{Hom}(T',G)$ (identified to those homomorphisms trivial on $N$), and $\mathrm{Hom}(T',G)$ is clopen in $\mathrm{Hom}(T,G)$ (indeed, let $Z$ have exponent $n$; then 1 is an isolated point in the set $K$ of elements $g\in G$ such that $g^n=1$ and $\mathrm{Hom}(T',G)$ is the fiber of $(1,\dots,1)$ for the mapping $\mathrm{Hom}(T,G)\to K^Z$ mapping $f$ to $z\mapsto f(z)$). Thus $\mathrm{Hom}(H,G)$ has infinitely many components (the proof even shows that it has a continuous map onto an infinite discrete set).
Conversely, suppose $H$ semisimple.
Consider the map $L:\mathrm{Hom}(H,G)\to\mathcal{L}(\mathfrak{h},\mathfrak{g})$ mapping $f$ to its tangent map between Lie algebras ($\mathcal{L}(\mathfrak{h},\mathfrak{g})$ denoting the whole space of linear maps). Then $L$ is injective. Since $L(f)$ is locally conjugate to $f$ by the exponential map, $L$ is continuous, and actually is a homeomorphism to its image (because if $L(f_n)$ tends to $L(f)$, then in a given compact neighborhood of 1 we have $f_n$ tending to $f$ uniformly, and this implies that $f_n$ tends to $f$ uniformly on all of $H$).
If $H$ is simply connected, then the image of $f$ is equal to the set of Lie algebra homomorphisms $\mathrm{Hom}(\mathfrak{h},\mathfrak{g})$, which is Zariski closed, and hence has finitely many components in the ordinary topology. In general (still assuming $H$ semisimple), let $\tilde{H}$ be its universal covering and $Z$ the (finite) kernel of $\tilde{H}\to H$. Recalling that the exponential is surjective, write $Z=\exp(Z')$ for some finite subset $Z'$ of $\mathfrak{h}$; then the image of $L$ is the set of elements $u$ in $\mathrm{Hom}(\mathfrak{h},\mathfrak{g})$ such that for all $z\in Z'$ we have $u(\exp(z))=1$. Let $\mathfrak{g}_1$ be the set of elements $x$ in $\mathfrak{g}$ such that $\exp(x)=1$: then we have the restatement: the image of $L$ is
$$\{f\in \mathrm{Hom}(\mathfrak{h},\mathfrak{g}): f(Z')\subset \mathfrak{g}_1\}.$$
Fix a faithful continuous linear representation of $G$ (in $\mathrm{SO}(n)$ for some $n$), so that we have a faithful representation of $\mathfrak{g}$ (by antisymmetric matrices): then $\mathfrak{g}_1$ is the set of elements in $\mathfrak{g}$ whose eigenvalues are in $2i\pi\mathbf{Z}$. Note that as soon as $G\neq 1$, it has infinitely many connected components (since all these eigenvalues can be achieved, using a 1-parameter subgroup whose image in $G$ is a 1-dimensional torus).
Nevertheless, observe that (still assuming $H$ compact semisimple) $\mathrm{Hom}(\mathfrak{h},\mathfrak{g})$ is a compact subset of the vector space $\mathcal{L}(\mathfrak{h},\mathfrak{g})$. Indeed, $\mathfrak{h}$ admits a basis $(e_j)$ such that each $e_j$ belongs to some subalgebra $\mathfrak{h}_j$ isomorphic to $\mathfrak{so}(3)$, such that some 2-dimensional complex representation of $\mathfrak{h}_i$ maps $e_j$ to the diagonal matrix $(i,-i)$ (where $i^2=-1$). It follows from representation theory of $\mathfrak{sl}_2$ that for every $n$-dimensional complex representation $\rho$ of $\mathfrak{h}_j$ (and hence of $\mathfrak{h}$) maps $e_j$ to an element with eigenvalues in $\{-i(n-1),\dots,i(n-1)\}$. Thus the trace of $-\rho(e_j)^2$ is $\le (n-1)^2n$. Since $-\mathrm{trace}(XY)$ is a definite positive symmetric bilinear form on antisymmetric matrices, this shows that $\mathrm{Hom}(\mathfrak{h},\mathfrak{g})$ is bounded, hence compact.
In particular, there exists $n_0$ such that eigenvalues of $f(z)$ for all $z\in Z'$ and $f\in\mathrm{Hom}(\mathfrak{h},\mathfrak{g})$ are in the interval $[-2i\pi n_0,2i\pi n_0]\subset i\mathbf{R}$. Thus the image by $L$ of $\mathrm{Hom}(H,G)$ is the set of $f\in\mathrm{Hom}(\mathfrak{h},\mathfrak{g})$ such that for every $z\in Z'$ we have $\prod_{k=-n_0}^{n_0} (f(z)-2i\pi k)=0$. This is a Zariski-closed subset, and hence has finitely many connected components in the ordinary topology, and hence so does the homeomorphic space $\mathrm{Hom}(H,G)$.