This is false for elliptic curves over fields of the second type that you describe, and probably over many other types of fields. For consistency with common notation, let's swap the roles of $\ell$ and $p$ and set $k = \mathbf F_{q^{\ell^\infty}}$ for a suitable power $q$ of $p$. We don't need to look very far:

**Lemma.** *Let $p > 3$ be a prime, let $\ell > p$ be prime, and let $k = \mathbf F_{q^{\ell^\infty}}$ where $q = p^2$. If $E$ is a supersingular curve defined over $\mathbf F_p$, then $E(k)$ has no $\ell$-torsion. In particular, it is not $\ell$-divisible.*

In fact, virtually the same argument shows that the analogous result for $\mathbf G_m$ is also false, contrary to what you claim. The problem is that you can't use Kummer theory if you don't have $\ell$-th roots of unity. An interesting variant would be under the further assumption that $E(k)$ contains full $\ell$-torsion (e.g. if $k$ contains an algebraically closed field).

*Proof.* The characteristic polynomial of Frobenius on $H^1_{\text{ét}}(E_{\bar k},\mathbf Q_{\ell'})$ for any prime $\ell' \neq p$ (possibly $\ell' = \ell$) is $x^2-p$; see for instance Exercise 5.10 in Silverman. Hence the eigenvalues of Frobenius are $\pm\sqrt{p}$, so the $q^n$-power Frobenius acts by scalar multiplication by $p^n$. From Weil's theorem, we get
$$\lvert E(\mathbf F_{q^n})\rvert = q^n-2p^n+1 =  (p^n-1)^2.$$
(Note that this agrees with $\lvert \mathbf G_m^2(\mathbf F_{p^n})\rvert$). By Fermat's little theorem, we get
$$\lvert E(\mathbf F_{q^{\ell^n}})\rvert = \big(p^{\ell^n}-1\big)^2 \equiv (p-1)^2 \pmod \ell.$$
This is nonzero since $0 < p-1 < \ell$, proving the first claim. The second follows since $E(k)$ is a nontrivial torsion group: if $P \in E(k)$ is some $n$-torsion element for $n > 1$, then $\ell$-divisibility would produce an $\ell n$-torsion element and hence an $\ell$-torsion element, which we saw is impossible. $\square$

To get a counterexample to the main question, you can of course take $C = E \setminus \{O\}$.