I'm thinking about the following argument. It's not entirely precise, I will be grateful for comments how to improve it.
Suppose such an $X$ is toric. Because the exceptional curves $E_i$ do not move inside $X$, they are fixed by the torus action. Therefore after blowing them down, the torus still acts and their images are fixed points. But there are only 3 fixed points of the torus action on $\mathbb{P}^2$!