Let $Y = P^2 \setminus ${two of its fixed points}, and $X = Y$. Then the fiber over $y$ is a point, but $Y$ is not contractible, I don't think.
I'm pretty sure your "has one fixed point" isn't the condition you want, but rather, "every $\lim_{z\to 0} z\cdot x$ exists and is $y$".