Let $K$ be the Galois closure of $F$. Since $F$ has prime discriminant, say $p$, the extension $K/\mathbb{Q}(\sqrt{p})$ is an unramified extension. In particular the ramification degree of $p$ in $K$ is 2, so it is impossible to have a sub-extension with ramification at $p$ not equal to $2$ or $1$.
Added to adress T.B comments below: What I said above can be generalized to the case when $F$ has fundamental discriminant $d$, i.e. if $p$ is a prime dividing $d$ it must happen that $pO_{F}=\mathcal{P}^2\mathcal{Q}$ for some $\mathcal{P} \neq \mathcal{Q}$. The proof is exactly the same as the above argument.
"I actually happened to compute the discriminant for $K$ and it turns out to be $p^3$ in all the cases I've done the calculations (assuming discriminant of $F$ is $p$). Do you think this is a coincidence or is it true?" Yes it is true, even in the general case that $disc(F)=d$ is fundamental. The point is that $K/\mathbb{Q}(\sqrt{d})$ is unramified, in other words its relative discriminant is trivial. Hence, $$disc(K/\mathbb{Q})=disc^{3}(\mathbb{Q}(\sqrt{d})/\mathbb{Q})N_{\mathbb{Q}(\sqrt{d})/\mathbb{Q})}(disc(K/\mathbb{Q}(\sqrt{d})))=d^3.$$
