How about this:
As you assumed, $P$ has terminal singularities, which means that the singular locus of $P$ has codimension at least 3 (see Corollary 5.18 in Kollár-Mori's book). Hence, the singular locus has codimension at most 2 in $X$. We can consider everything outside the singular locus because we work on the level of divisors of $X$. Hence $P$ is just smooth and $X$ is Cartier and everything is true for you.