A more general version of this statement was shown by Kimura in Acta Arith. (2003). His corollary gives $\gg \sqrt{X}/\log X$ such fields ${\Bbb Q}(\sqrt{-d})$ with $d\le X$, and also allows you to add further
splitting conditions. There is an extensive literature on divisibility and indivisibility of class numbers and Kimura's paper has many other relevant references.

**Update** There is also an erratum to Kimura's paper -- it seems that he needs the existence of one such field to get a lower bound for the number of such fields; for large $p$ this is guaranteed by work of Horie. In the meantime a paper of Wiles has addressed the general version of this question, and there is a quantification of Wiles's result due to Beckwith.