I find this question extremely interesting. I have two small observations.

First, following up on my comment, the answer is definitely no in the case that you allow real coefficients and want the answer in finite time. It seems natural to suppose that we are given the form of the two polynomials, and then also given the coefficients as oracles. Perhaps we are given an infinite sequence of rational approximations to them, with a known rate of convergence. The difficulty is that it is impossible in principle to compute in finite time whether two oracles are equal. (If they look the same so far, then you cannot say "they are equal" at any finite time, since a difference may arise at some later point that you never inspected.) Similarly, it is impossible in principle to determine if an oracle is $0$ or not in finite time.

Suppose we could decide your problem. Now, given a reals $a$, construct the two polynomials $p(x)=0$ and $q(x)=ax$. In the case that $a=0$, then the solution sets of these polynomials are homeomorphic, since the polynomials are both the zero polynomial. But in the case that $a\neq 0$, then they are not homemorphic, since every $x$ solves $p$ but only $x=0$ solves $q$. Thus, the zero-test problem reduces to your problem, and so your problem is not decidable.

But as I mentioned in my comment, I think in the case of real coefficients we didn't really expect to get an answer in finite time. This is why it is natural to consider the question of what happens with rational coefficients, where the algorithm has full access to the entire system.

Here, I don't have an anwer, but merely offer the observation that if somehow the question is expressible in the language of the first order structure $\langle R,+,\cdot,0,1,\lt\rangle$, that is, in the language of real-closed fields, then it will be decidable by Tarski's theorem, which asserts that the theory of this structure is decidable. We have a computable procedure that answers any first order inquiry about this structure. But I'm not sure that your problem is expressible in this language, and I suspect it isn't. So meanwhile I will wait for the algebraic geometers to settle it.

werehomeomorphic or that theyweren'thomeomorphic, then there would be a decision procedure, namely, go and look for a proof one way or the other and output the corresponding answer. That is, if the problem is undecidable, then there is a pair of rational polynomials such that it is neither provable nor refutable in ZFC that the solution sets are homeomorphic. (This would be an existence proof that there is an explicit example.) $\endgroup$ – Joel David Hamkins May 21 '10 at 19:05