Here is a link to Serre's paper: Analogues Kahleriennes de Certaines Conjectures de Weil. There seems to be some misunderstanding about what Serre actually proves implicit in the question, so I'll clarify it a bit, explain an easy analogue for some non-compact varieties, and indicate what should be true in general.
First of all, Serre's result is not for arbitrary compact Kahler manifolds. Rather, he starts with a smooth complex projective variety $X$, and an endomorphism $f: X\to X$, with an ample divisor $E$ so that $f^{-1}(E)$ is algebraically equivalent to $qE$ for some integer $q>0$. Then he shows that the eigenvalues of $f^*$ acting on $H^r(X, \mathbb{C})$ have absolute value $q^{r/2}$. Contrary to the statement of the question, the algebraicity of $X$ is built into the very result, since it requires the existence of an ample divisor.
Serre doesn't explicitly define the zeta function of $(X, f)$, but by analogy to the Weil conjectures, one may define $$Z(X, f, t)=\prod_i \det(1-f^*t\mid H^i(X, \mathbb{C}))^{(-1)^{i+1}}.$$ Then this zeta function satisfies the desired "Riemann hypothesis," by the result above.
When looking for a non-compact analogue, the motto one should have in mind is that zeta functions should behave well under "cutting-and-pasting." For example, if $Z(X, t)$ is the zeta function from the Weil conjectures, where $X/\mathbb{F}_q$ is an arbitrary variety, and $Y\subset X$ is a closed subscheme, then $$Z(X, t)=Z(X\setminus Y, t) \cdot Z(Y, t).$$
So here's an analogue of this fact in the setting of smooth quasi-projective varieties. Suppose $X$ is a smooth projective variety over $\mathbb{C}$, and $f: X\to X$ is a self-map satisfying the conditions of Serre's result. Suppose $Y\subset X$ is a smooth closed subvariety, so that $f(Y)\subset Y$. Then $Y, f|_Y$ also satisfy the conditions of the theorem, and so $Z(X, f, t)$ and $Z(Y, f|_Y, t)$ satisfy the "Riemann Hypothesis." Now suppose $f(X\setminus Y)\subset X\setminus Y$ as well, and $f|_{X\setminus Y}$ is proper. Then we may define $$Z(X\setminus Y, f|_{X\setminus Y}, t)=\prod_i \det(1-f|_{X\setminus Y}^*t\mid H^i_c(X\setminus Y, \mathbb{C}))^{(-1)^{i+1}}.$$
Here $H^i_c(X, \mathbb{C})$ denotes the cohomology of $X$ with compact support. Does this zeta function satisfy some kind of Riemann hypothesis? Well, from the long exact sequence relating the compactly supported cohomology of $X\setminus Y$ to that of $X$ and $Y$, we have that $$Z(X\setminus Y, f|_{X\setminus Y}, t)=Z(X, f, t)/Z(Y, f|_Y, t).$$
So certainly all of the poles and zeroes of $Z(X\setminus Y, f|_{X\setminus Y}, t)$ have absolute value $q^{-r/2}$ for some integer $r$. This gives a sort of "Riemann hypothesis" for quasi-projective varieties admitting a particularly nice compactification. What you'll notice immediately though is that the $r$ does not match up with the cohomological degree, as it does in the compact case---there is some "slippage" coming from the boundary map in the long exact sequence. Rather, the $r$ is related to the "weight filtration" on the cohomology of $X\setminus Y$.
Now suppose $U$ is an arbitrary quasiprojective variety, and $f: U\to U$ is a proper map. $U$ might not have the ridiculously nice compactification we need to run the above argument, but the zeta function defined using cohomology with compact support still makes sense. I doubt that it will in general satisfy a "Riemann hypothesis" of the type you're looking for, since the condition on ample divisors may not makes sense. What is true, though, is that if $U$ admits a smooth compactification $X$ so that $f$ extends to a map $g: X\to X$ satisfying the conditions of Serre's result, so that $g(X\setminus U)\subset X\setminus U$, (where $X\setminus U$ is not necessarily smooth!) this zeta function will satisfy a "Riemann hypothesis." Namely, the zeroes and poles of $Z(U, f, t)$ will be of the form $q^{-r/2}$, with the $r$ coming from the weight filtration on the compactly supported cohomology of $U$.
To see this, one may imitate Serre's argument using the mixed Hodge structure on the cohomology of $X\setminus Z$. I don't immediately see how to give an analogue of Serre's condition on the existence of the ample divisor $E$ for $U$ without reference to a compactification, though there should be a way to do so; then one should be able to imitate Serre's argument, just working with the mixed Hodge structure on the compactly supported cohomology of $U$.
$ $
