This is ultimately an application of Lang's vanishing theorem for degree-1 Galois cohomology of connected algebraic groups over finite fields (applied to tori).
What follows may look complicated if you haven't worked much with tori, but to Serre in those days this sort of thing was bread and butter (and it is all "standard" material for work with the arithmetic aspects of algebraic groups).
Observe that $E := F \otimes_{\mathbf{Q}} F'$ is a subfield of $K$, and if $K/E$ is a finite extension of number fields that is unramified at all places over $\ell$ then the norm map $O_{K,\ell}^{\times} \rightarrow O_{E,\ell}^{\times}$ is surjective (where $O_{E,\ell} = \prod_{v|\ell} O_{E,v}$) since it is very classical that the norm map between integral unit groups relative to an unramified extension of non-archimedean local fields is surjective.
Thus, $K$ now drops out of the picture and the problem is a special case of the following setup. Let $k$ be a global field, and $F$ and $F'$ two finite etale $k$-algebras (e.g., $k = \mathbf{Q}$ and $F$ and $F'$ two number fields). Let $E = F \otimes_k F'$, another finite etale $k$-algebra. Then the claim is that for all but finitely many non-archimedean places $v$ of $k$, the map $$O_{E,v}^{\times} \rightarrow O_{F,v}^{\times} \times O_{F',v}^{\times}$$
defined by $t \mapsto ({\rm{N}}_{E/F}(t), {\rm{N}}_{E/F'}(t))$ has image equal to the group $H$ of pairs $(u,u')$ for which ${\rm{N}}_{F/k}(u) = {\rm{N}}_{F'/k}(u')$. (Here, $O_{E,v}$ is the direct product of valuation rings of factor fields of
$E \otimes_k k_v$, etc.) This is ultimately a special case of a well-known general fact about homomorphisms between tori over global fields, as follows.
Consider the Weil-restriction $k$-tori $T = {\rm{R}}_{E/k}({\rm{GL}}_1)$,
$S = {\rm{R}}_{F/k}({\rm{GL}}_1)$, and $S' = {\rm{R}}_{F'/k}({\rm{GL}}_1)$, so the maximal compact subgroup of $T(O_{k,v})$ is $O_{E,v}^{\times}$ and similarly for $S$ and $S'$ using $F$ and $F'$ respectively. The $k$-algebra inclusions of $F$ and $F'$ into $E$ induce norm homomorphisms of $k$-tori $T \rightrightarrows S, S'$, and the resulting map $N:T \rightarrow S \times S'$ applied to $k_v$-points induces the map of interest between maximal compact subgroups.
The first thing to check is that $N$ has torus kernel and image (in the sense of algebraic groups) equal to the $k$-torus of pairs $(s,s')$ such that ${\rm{N}}_{F/k}(s) = {\rm{N}}_{F'/k}(s')$, which is a general claim we make with $k$ an arbitrary field (has nothing to do with number theory). This is exactly the statement that the induced map between geometric character groups has torsion-free cokernel and has kernel equal to the $\mathbf{Z}$-span of the character $\chi:(s,s') \mapsto {\rm{N}}_{F/k}(s)/{\rm{N}}_{F'/k}(s')$. Because our setup has been given in the appropriate generality, we may assume $k$ is separably closed, so then $F$ and $F'$ are split $k$-algebras, say $F = k^I$ and $F' = k^J$ for non-empty finite sets $I$ and $J$.
Then $E = k^{I \times J}$ with the $k$-subalgebra inclusions of $F$ and $F'$ into $E$ respectively dual to the projections $I \times J \rightrightarrows I, J$. Thus, the associated map via $N$ between character groups is $\mathbf{Z}^I \times \mathbf{Z}^J \rightarrow \mathbf{Z}^{I \times J}$ induced by $((n_i), (m_j)) \mapsto (n_i + m_j)$. This has kernel consisting of pairs of constant sequences $n_i = c$ and $m_j = -c$ for $c \in \mathbf{Z}$, and for $c = 1$ this is $\chi$. As for the cokernel being torsion-free, we just have to check for a prime $p$ that if
$t_{ij} \in \mathbf{Z}$ and $p t_{ij} = n_i + m_j$ for some $n_i, m_j \in \mathbf{Z}$ then we can choose such $n_i$ and $m_j$ to be divisible by $p$.
But $n_i \equiv -m_j \bmod p$ for all $i, j$, so for $c = n_1$ we
have $n_i \equiv c \bmod p$ and $m_j \equiv -c \bmod p$ for all $i, j$.
We can then replace $n_i$ with $n_i - c$ and $m_j$ with $m_j + c$ for all $i, j$
to get what we need.
Finally we come to the part with arithmetic content: if $k$ is a global field and $f:T \rightarrow T''$ is a surjection between $k$-tori (such as $N$ mapping onto its indicated image torus above!) with kernel equal to a $k$-torus $T'$ then for all but finitely many non-archimedean places $v$ of $k$ the induced map $T(k_v) \rightarrow T''(k_v)$ restricts to a surjection between maximal compact subgroups. In fact, this holds for any place $v$ at which the geometric character lattice of $T$ (and hence of $T''$ and $T'$) is unramified.
Indeed, for such $v$ the Galois-descent construction of the tori may be carried out not only through a finite unramified extension of $k_v$ but even through the corresponding finite etale extension of valuation rings. As such, we get a short exact sequence of $O_{k_v}$-tori
$$1 \rightarrow T'_v \rightarrow T_v \rightarrow T''_v \rightarrow 1.$$
The map $T_v(O_v) \rightarrow T''(O_v)$ is surjective because the obstructions are $T'_v$-torsors over $O_v$, all of which vanish by Lang's theorem over the finite residue field (applied to the torus reduction of $T'_v$) and by Hensel's Lemma (to lift a rational point over the residue field to an $O_v$-point).
So it remains to show that for any $O_v$-torus $S$ (such as $T''_v$), $S(O_v)$ is the maximal compact subgroup of $S(k_v)$. This is easily pulled down from the split case over a finite unramified extension of $k_v$ that splits $S$.
