Here is a link to Serre's paper: Analogues Kahleriennes de Certaines Conjectures de Weil. There seems to be some misunderstanding about what Serre actually proves implicit in the question, so I'll clarify it a bit, explain an easy analogue for some non-compact varieties, and indicate what should be true in general.

First of all, Serre's result is *not* for arbitrary compact Kahler manifolds. Rather, he starts with a smooth complex projective variety $X$, and an endomorphism $f: X\to X$, with an ample divisor $E$ so that $f^{-1}(E)$ is algebraically equivalent to $qE$ for some integer $q>0$. Then he shows that the eigenvalues of $f^*$ acting on $H^r(X, \mathbb{C})$ have absolute value $q^{r/2}$. Contrary to the statement of the question, the algebraicity of $X$ is built into the very result, since it requires the existence of an ample divisor.

Serre doesn't explicitly define the zeta function of $(X, f)$, but by analogy to the Weil conjectures, one may define $$Z(X, f, t)=\prod_i \det(1-f^*t\mid H^i(X, \mathbb{C}))^{(-1)^{i+1}}.$$ Then this zeta function satisfies the desired "Riemann hypothesis," by the result above.

When looking for a non-compact analogue, the motto one should have in mind is that zeta functions should behave well under "cutting-and-pasting." For example, if $Z(X, t)$ is the zeta function from the Weil conjectures, where $X/\mathbb{F}_q$ is an arbitrary variety, and $Y\subset X$ is a closed subscheme, then $$Z(X, t)=Z(X\setminus Y, t) \cdot Z(Y, t).$$

So here's an analogue of this fact in the setting of smooth quasi-projective varieties. Suppose $X$ is a smooth projective variety over $\mathbb{C}$, and $f: X\to X$ is a self-map satisfying the conditions of Serre's result. Suppose $Y\subset X$ is a smooth closed subvariety, so that $f(Y)\subset Y$. Then $Y, f|_Y$ also satisfy the conditions of the theorem, and so $Z(X, f, t)$ and $Z(Y, f|_Y, t)$ satisfy the "Riemann Hypothesis." Now suppose $f(X\setminus Y)\subset X\setminus Y$ as well, and $f|_{X\setminus Y}$ is proper. Then we may define $$Z(X\setminus Y, f|_{X\setminus Y}, t)=\prod_i \det(1-f|_{X\setminus Y}^*t\mid H^i_c(X\setminus Y, \mathbb{C}))^{(-1)^{i+1}}.$$

Here $H^i_c(X, \mathbb{C})$ denotes the cohomology of $X$ with compact support. Does this zeta function satisfy some kind of Riemann hypothesis? Well, from the long exact sequence relating the compactly supported cohomology of $X\setminus Y$ to that of $X$ and $Y$, we have that $$Z(X\setminus Y, f|_{X\setminus Y}, t)=Z(X, f, t)/Z(Y, f|_Y, t).$$

So certainly all of the poles and zeroes of $Z(X\setminus Y, f|_{X\setminus Y}, t)$ have absolute value $q^{-r/2}$ for some integer $r$. This gives a sort of "Riemann hypothesis" for quasi-projective varieties admitting a particularly nice compactification. What you'll notice immediately though is that the $r$ does not match up with the cohomological degree, as it does in the compact case---there is some "slippage" coming from the boundary map in the long exact sequence. Rather, the $r$ is related to the "weight filtration" on the cohomology of $X\setminus Y$.

Now suppose $U$ is an arbitrary quasiprojective variety, and $f: U\to U$ is a proper map. $U$ might not have the ridiculously nice compactification we need to run the above argument, but the zeta function defined using cohomology with compact support still makes sense. I doubt that it will in general satisfy a "Riemann hypothesis" of the type you're looking for, since the condition on ample divisors may not makes sense. What is true, though, is that if $U$ admits a smooth compactification $X$ so that $f$ extends to a map $g: X\to X$ satisfying the conditions of Serre's result, so that $g(X\setminus U)\subset X\setminus U$, (where $X\setminus U$ is not necessarily smooth!) this zeta function will satisfy a "Riemann hypothesis." Namely, the zeroes and poles of $Z(U, f, t)$ will be of the form $q^{-r/2}$, with the $r$ coming from the weight filtration on the compactly supported cohomology of $U$.

To see this, one may imitate Serre's argument using the mixed Hodge structure on the cohomology of $X\setminus Z$. I don't immediately see how to give an analogue of Serre's condition on the existence of the ample divisor $E$ for $U$ without reference to a compactification, though there should be a way to do so; then one should be able to imitate Serre's argument, just working with the mixed Hodge structure on the compactly supported cohomology of $U$.

$ $

Jeg gider ikkefigure out whether the analogue for Stein manifolds is vacuously true or false, as there almost all cohomology groups are zero, but in any case it's not interesting. For non-compact manifolds all cohomology groups might be zero or infinite dimensional, so one needs to tread carefully in generalizing RH. A tractable question might be if there is an analogue of RH for copmact non-Kahler manifolds, like the Hopf surface or the Iwasawa manifold? – Gunnar Þór Magnússon Oct 9 '11 at 17:37zeta function of the map. I could imagine how the noncompact version might look, but I'd prefer not to speculate too much right now. – Donu Arapura Oct 9 '11 at 19:57