My previous answer addressed character sums for all integers, not primes. Apologies for reading the question too quickly.

Concerning the real question over primes, as the commenters said, it is addressed by the prime number theorem for the given quadratic character. First, one can replace $d$ by its square-free part ($d$ divided by its largest square divisor) at the cost of an error term of $O(\tau(d))$. Then, the new quadratic character will be primitive (with conductor dividing $4d$), and one can apply Theorem 5.27 in Iwaniec-Kowalski: Analytic number theory. This theorem addresses the analogous sum over all prime powers with the usual von Mangoldt weights $\Lambda(n)$. One can restrict this sum to primes at the cost of an error term of $O(\tau(d)X^{1/2})$, and then one can remove the remaining $\log p$ weights by partial summation. The upshot is that the sum can be bounded by the right hand side of (5.70) in Iwaniec-Kowalski (or even by the same divided by $\log X$). In particular, the size of the sum depends on the Landau-Siegel zero for $d$ if this zero exists (but hopefully it does not exist).

For a quick bound, see (5.79) in the same book. It implies that the sum is $\ll_A\sqrt{d} X(\log X)^{-A}$ for any $A>0$.