This is false for elliptic curves over fields of the second type that you describe, and probably over many other types of fields. For consistency with common notation, let's swap the roles of $\ell$ and $p$ and set $k = \mathbf F_{q^{\ell^\infty}}$ for a suitable power $q$ of $p$. We don't need to look very far:

Lemma 1. Let $p > 3$ be a prime, let $\ell > p$ be prime, and let $k = \mathbf F_{q^{\ell^\infty}}$ where $q = p^2$. If $E$ is a supersingular curve defined over $\mathbf F_p$, then $E(k)$ has no $\ell$-torsion. In particular, it is not $\ell$-divisible.

In fact, virtually the same argument shows that the analogous result for $\mathbf G_m$ is also false, contrary to what you claim. The problem is that Kummer theory only works if $k$ contains the $\ell$-th roots of unity. But very much like the $\mathbf G_m$ case, there is a positive result if you assume that $E(k)$ contains full $\ell$-torsion; see Lemma 2.

Proof of Lemma 1. The characteristic polynomial of Frobenius on $H^1_{\text{ét}}(E_{\bar k},\mathbf Q_{\ell'})$ for any prime $\ell' \neq p$ (possibly $\ell' = \ell$) is $x^2-p$; see for instance Exercise 5.10 in Silverman. Hence the eigenvalues of Frobenius are $\pm\sqrt{p}$, so the $q^n$-power Frobenius acts by scalar multiplication by $p^n$. From Weil's theorem, we get $$\lvert E(\mathbf F_{q^n})\rvert = q^n-2p^n+1 = (p^n-1)^2.$$ (Note that this agrees with $\lvert \mathbf G_m^2(\mathbf F_{p^n})\rvert$). By Fermat's little theorem, we get $$\lvert E(\mathbf F_{q^{\ell^n}})\rvert = \big(p^{\ell^n}-1\big)^2 \equiv (p-1)^2 \pmod \ell.$$ This is nonzero since $0 < p-1 < \ell$, proving the first claim. The second follows since $E(k)$ is a nontrivial torsion group: if $P \in E(k)$ is some $n$-torsion element for $n > 1$, then $\ell$-divisibility would produce an $\ell n$-torsion element and hence an $\ell$-torsion element, which we saw is impossible. $\square$

To get a counterexample to the main question, you can of course take $C = E \setminus \{O\}$.

Lemma 2. Let $k$ be a field and $\ell$ a prime invertible in $k$, such that all finite separable extensions of $k$ have degree prime to $\ell$. Let $A$ be an abelian variety over $k$ such that $A(k)$ has full $\ell$-torsion, and let $P \in A(k)$. Then all points $Q \in A(k^{\text{sep}})$ with $[\ell] Q = P$ are defined over $k$.

Proof. The scheme-theoretic preimage of $P$ under $[\ell] \colon A \to A$ is naturally an étale $A[\ell]$-torsor over $k = \kappa(P)$, and the assumption on $A$ implies that the étale group scheme $A[\ell] \to \operatorname{Spec} k$ is constant. Then $H^1(k,A[\ell]) = \operatorname{Hom}^{\text{cts}}(\operatorname{Gal}(k^{\text{sep}}/k),A[\ell])$ is trivial by the assumption on $k$, so the torsor is trivial. $\square$.

(Again, this is the same way that you prove Kummer theory.)

Inspired by the $\mathbf G_m$ case (Kummer theory), here is a positive result if you assume that $E(k)$ contains full $\ell$-torsion:

Lemma. Let $k$ be a field and $\ell$ a prime invertible in $k$, such that all finite separable extensions of $k$ have degree prime to $\ell$. Let $A$ be an abelian variety over $k$ such that $A(k)$ has full $\ell$-torsion, and let $P \in A(k)$. Then all points $Q \in A(k^{\text{sep}})$ with $[\ell] Q = P$ are defined over $k$.

Proof. The scheme-theoretic preimage of $P$ under $[\ell] \colon A \to A$ is naturally an étale $A[\ell]$-torsor over $k = \kappa(P)$, and the assumption on $A$ implies that the étale group scheme $A[\ell] \to \operatorname{Spec} k$ is constant. Then $H^1(k,A[\ell]) = \operatorname{Hom}^{\text{cts}}(\operatorname{Gal}(k^{\text{sep}}/k),A[\ell])$ is trivial by the assumption on $k$, so the torsor is trivial. $\square$.

For $\mathbf G_m$, the only alternative is that $k^\times$ has no $\ell$-torsion, in which case it is trivially $\ell$-divisible (as $[\ell]$ is invertible on $A(k)$). But on abelian varieties, there are intermediate cases with some but not all $\ell$-torsion, on which I have little to say at the moment.

This is false for elliptic curves over fields of the second type that you describe, and probably over many other types of fields. For consistency with common notation, let's swap the roles of $\ell$ and $p$ and set $k = \mathbf F_{q^{\ell^\infty}}$ for a suitable power $q$ of $p$. We don't need to look very far:

Lemma. Let $p > 3$ be a prime, let $\ell > p$ be prime, and let $k = \mathbf F_{q^{\ell^\infty}}$ where $q = p^2$. If $E$ is a supersingular curve defined over $\mathbf F_p$, then $E(k)$ has no $\ell$-torsion. In particular, it is not $\ell$-divisible.

In fact, virtually the same argument shows that the analogous result for $\mathbf G_m$ is also false, contrary to what you claim. The problem is that you can't use Kummer theory if you don't have $\ell$-th roots of unity. An interesting variant would be under the further assumption that $E(k)$ contains full $\ell$-torsion (e.g. if $k$ contains an algebraically closed field).

Proof. The characteristic polynomial of Frobenius on $H^1_{\text{ét}}(E_{\bar k},\mathbf Q_{\ell'})$ for any prime $\ell' \neq p$ (possibly $\ell' = \ell$) is $x^2-p$; see for instance Exercise 5.10 in Silverman. Hence the eigenvalues of Frobenius are $\pm\sqrt{p}$, so the $q^n$-power Frobenius acts by scalar multiplication by $p^n$. From Weil's theorem, we get $$\lvert E(\mathbf F_{q^n})\rvert = q^n-2p^n+1 = (p^n-1)^2.$$ (Note that this agrees with $\lvert \mathbf G_m^2(\mathbf F_{p^n})\rvert$). By Fermat's little theorem, we get $$\lvert E(\mathbf F_{q^{\ell^n}})\rvert = \big(p^{\ell^n}-1\big)^2 \equiv (p-1)^2 \pmod \ell.$$ This is nonzero since $0 < p-1 < \ell$, proving the first claim. The second follows since $E(k)$ is a nontrivial torsion group: if $P \in E(k)$ is some $n$-torsion element for $n > 1$, then $\ell$-divisibility would produce an $\ell n$-torsion element and hence an $\ell$-torsion element, which we saw is impossible. $\square$

To get a counterexample to the main question, you can of course take $C = E \setminus \{O\}$.

This is false for elliptic curves over fields of the second type that you describe, and probably over many other types of fields. For consistency with common notation, let's swap the roles of $\ell$ and $p$ and set $k = \mathbf F_{q^{\ell^\infty}}$ for a suitable power $q$ of $p$. We don't need to look very far:

Lemma 1. Let $p > 3$ be a prime, let $\ell > p$ be prime, and let $k = \mathbf F_{q^{\ell^\infty}}$ where $q = p^2$. If $E$ is a supersingular curve defined over $\mathbf F_p$, then $E(k)$ has no $\ell$-torsion. In particular, it is not $\ell$-divisible.

In fact, virtually the same argument shows that the analogous result for $\mathbf G_m$ is also false, contrary to what you claim. The problem is that Kummer theory only works if $k$ contains the $\ell$-th roots of unity. But very much like the $\mathbf G_m$ case, there is a positive result if you assume that $E(k)$ contains full $\ell$-torsion; see Lemma 2.

Proof of Lemma 1. The characteristic polynomial of Frobenius on $H^1_{\text{ét}}(E_{\bar k},\mathbf Q_{\ell'})$ for any prime $\ell' \neq p$ (possibly $\ell' = \ell$) is $x^2-p$; see for instance Exercise 5.10 in Silverman. Hence the eigenvalues of Frobenius are $\pm\sqrt{p}$, so the $q^n$-power Frobenius acts by scalar multiplication by $p^n$. From Weil's theorem, we get $$\lvert E(\mathbf F_{q^n})\rvert = q^n-2p^n+1 = (p^n-1)^2.$$ (Note that this agrees with $\lvert \mathbf G_m^2(\mathbf F_{p^n})\rvert$). By Fermat's little theorem, we get $$\lvert E(\mathbf F_{q^{\ell^n}})\rvert = \big(p^{\ell^n}-1\big)^2 \equiv (p-1)^2 \pmod \ell.$$ This is nonzero since $0 < p-1 < \ell$, proving the first claim. The second follows since $E(k)$ is a nontrivial torsion group: if $P \in E(k)$ is some $n$-torsion element for $n > 1$, then $\ell$-divisibility would produce an $\ell n$-torsion element and hence an $\ell$-torsion element, which we saw is impossible. $\square$

To get a counterexample to the main question, you can of course take $C = E \setminus \{O\}$.

Lemma 2. Let $k$ be a field and $\ell$ a prime invertible in $k$, such that all finite separable extensions of $k$ have degree prime to $\ell$. Let $A$ be an abelian variety over $k$ such that $A(k)$ has full $\ell$-torsion, and let $P \in A(k)$. Then all points $Q \in A(k^{\text{sep}})$ with $[\ell] Q = P$ are defined over $k$.

Proof. The scheme-theoretic preimage of $P$ under $[\ell] \colon A \to A$ is naturally an étale $A[\ell]$-torsor over $k = \kappa(P)$, and the assumption on $A$ implies that the étale group scheme $A[\ell] \to \operatorname{Spec} k$ is constant. Then $H^1(k,A[\ell]) = \operatorname{Hom}^{\text{cts}}(\operatorname{Gal}(k^{\text{sep}}/k),A[\ell])$ is trivial by the assumption on $k$, so the torsor is trivial. $\square$.

(Again, this is the same way that you prove Kummer theory.)