Let $Y = P^2 \setminus ${two of its fixed points}, and $X = Y$.
Then the fiber over $y$ is a point, but $Y$ is not contractible, I don't think.

I'm pretty sure your "has one fixed point" isn't the condition you want,
but rather, "every $\lim_{z\to 0} z\cdot x$ exists and is $y$".