Having resolved my ignorance concerning surface groups I can now answer question 1 negatively (or at least some formulation thereof). It is impossible if $Y$ is an oriented surface of genus at least $2$.
Suppose that $f: Y \to Y$ is a self map of the surface such that the eigenvalues of $f^*$ acting on each $H^i(Y)$ are all nonzero (otherwise we can't "detect" the betti numbers), and such that $H^i(Y)$ and $H^j(Y)$ do not have eigenvalues of common magnitude for $i \neq j$. Then in particular $f^*$ acts on $H^2(Y)$ nontrivially, say by multiplication by some integer $d$. This integer cannot be $\pm 1$ since then $H^0(Y)$ and $H^2(Y)$ would contain eigenvectors with eigenvalues of equal magnitude.
Consider the subgroup $H = f_*(\pi_1(Y))$ inside $G = \pi_1(Y)$. If this had infinite index, then $f$ would lift to a map to some infinite covering of $Y$, so it would induce a trivial map of $H^2$. So $H$ has finite index in $G$. Let $X \to Y$ be the corresponding covering space. Then $\pi_1(X)$ is a quotient of $\pi_1(Y)$, hence its abelianization has rank $\leq 2g$ where $g$ is the genus of $Y$. This implies that $X$ is a closed surface of genus at most $g$. But its Euler characteristic is precisely $[G:H]$ times the Euler characteristic of $Y$, so $X = Y$. Thus $f$ induces a surjection on $\pi_1(Y)$. By the post cited above, $f$ actually induces an isomorphism on $\pi_1(Y)$, so it is a homotopy equivalence. In particular, $d = \pm 1$, contrary to assumption.
After writing this it occurs to me that you might object to me ruling out the case $d = -1$... At any rate, this shows that the eigenvalues can't ever look like they do in the case of the Riemann hypothesis, with magnitude $q^{i/2}$ on $H^i$ for some $q>1$.
