This can be done if $\ell \equiv 1 \pmod{4}$.

In the article here, Andrew Wiles proves that if $\ell \geq 5$ is a prime and $S$ is a finite set of primes written as a disjoint union $S = S_{-} \cup S_{0} \cup S_{+}$ with the properties that

$\bullet$ $q \in S_{-} \implies q \not\equiv 1 \pmod{\ell} \text{ and } q \not\equiv 3 \pmod{4}$, and

$\bullet$ $q \in S_{0} \implies q \not\equiv 1 \pmod{\ell}$, and

$\bullet$ $q \in S_{+} \implies q \not\equiv -1 \pmod{\ell}$,

then there is a fundamental discriminant $d < 0$ so that the class number of $\mathbb{Q}(\sqrt{d})$ is coprime to $\ell$ and $\left(\frac{d}{q}\right) = -1$ for $q \in S_{-}$, $\left(\frac{d}{q}\right) = 0$ for $q \in S_{0}$, and $\left(\frac{d}{q}\right) = 1$ for $q \in S_{+}$.

If $\ell \equiv 1 \pmod{4}$, we can apply this with $S_{-} = \{ 2, \ell \}$ and $S_{0} = S_{+} = \emptyset$. Wiles's result gives an imaginary quadratic field in which $2$ and $\ell$ are inert with class number indivisible by $\ell$.

Note: A quantitative version of Wiles's theorem (with the same restrictions on the sets $S_{-}$, $S_{0}$ and $S_{+}$) was proven by Olivia Beckwith in the article here.

Here's an elementary argument. For $\ell < 41$, $K = \mathbb{Q}(\sqrt{-163})$ works. For $\ell = 41$, $K = \mathbb{Q}(\sqrt{-3})$ works. Assume then that $\ell \geq 43$.

Choose an integer $1 \leq n \leq \ell - 1$ so that $\left(\frac{-n}{\ell}\right) = -1$ and let $k \in \{ 0, 1, 2, 3 \}$ be the unique integer so that $n+k \ell \equiv 3 \pmod{4}$. Write $-(n+k \ell) = d r^{2}$ with $d$ squarefree. Note that $d \equiv 1 \pmod{4}$ so $d$ is an odd fundamental discriminant. Moreover, $\left(\frac{d}{\ell}\right) = \left(\frac{-n}{\ell}\right) = -1$ and so $\ell$ is inert in $\mathbb{Q}(\sqrt{d})$.

The convexity bound for $L(1,\chi_{d})$ and Dirichlet's class number formula gives the bound $h(d) \leq \frac{2}{\pi} \sqrt{|d|} \log(|d|)$. Thus, $$ h(d) \leq \frac{2}{\pi} \sqrt{|d|} \log(|d|) < \frac{2}{\pi} \sqrt{4 \ell} \log(4 \ell), $$ and it is straightforward to see that this is $< \ell$ if $\ell \geq 43$. Hence $(\ell,h_{\mathbb{Q}(\sqrt{d})}) = 1$.

Note: In a previous version of this answer, I mention a result of Wiles here, which states that if $\ell \geq 5$ is a prime and $S$ is a finite set of odd primes written as a disjoint union $S = S_{-} \cup S_{0} \cup S_{+}$ with the properties that

$\bullet$ $q \in S_{-} \implies q \not\equiv 1 \pmod{\ell} \text{ and } q \not\equiv 3 \pmod{4}$, and

$\bullet$ $q \in S_{0} \implies q \not\equiv 1 \pmod{\ell}$, and

$\bullet$ $q \in S_{+} \implies q \not\equiv -1 \pmod{\ell}$,

then there is a fundamental discriminant $d < 0$ so that the class number of $\mathbb{Q}(\sqrt{d})$ is coprime to $\ell$ and $\left(\frac{d}{q}\right) = -1$ for $q \in S_{-}$, $\left(\frac{d}{q}\right) = 0$ for $q \in S_{0}$, and $\left(\frac{d}{q}\right) = 1$ for $q \in S_{+}$.

However, Wiles's result does not apply since the elements of $S_{-}$, $S_{0}$ and $S_{+}$ must be odd.