Having resolved my [ignorance][1] concerning surface groups I can now answer question 1 negatively (or at least some formulation thereof).  It is impossible if $Y$ is an oriented surface of genus at least $2$.

Suppose that $f: Y \to Y$ is a self map of the surface such that the eigenvalues of $f^*$ acting on each $H^i(Y)$ are all nonzero (otherwise we can't "detect" the betti numbers), and such that $H^i(Y)$ and $H^j(Y)$ do not have eigenvalues of common magnitude for $i \neq j$.  Then in particular $f^*$ acts on $H^2(Y)$ nontrivially, say by multiplication by some integer $d$.  This integer cannot be $\pm 1$ since then $H^0(Y)$ and $H^2(Y)$ would contain eigenvectors with eigenvalues of equal magnitude.

Consider the subgroup $H = f_*(\pi_1(Y))$ inside $G = \pi_1(Y)$.  If this had infinite index, then $f$ would lift to a map to some infinite covering of $Y$, so it would induce a trivial map of $H^2$.  So $H$ has finite index in $G$.  Let $X \to Y$ be the corresponding covering space.  Then $\pi_1(X)$ is a quotient of $\pi_1(Y)$, hence its abelianization has rank $\leq 2g$ where $g$ is the genus of $Y$.  This implies that $X$ is a closed surface of genus at most $g$.  But its Euler characteristic is precisely $[G:H]$ times the Euler characteristic of $Y$, so $X = Y$.  Thus $f$ induces a surjection on $\pi_1(Y)$.  By the post cited above, $f$ actually induces an isomorphism on $\pi_1(Y)$, so it is a homotopy equivalence.  In particular, $d = \pm 1$, contrary to assumption.  

After writing this it occurs to me that you might object to me ruling out the case $d = -1$...  At any rate, this shows that the eigenvalues can't ever look like they do in the case of the Riemann hypothesis, with magnitude $q^{i/2}$ on $H^i$ for some $q>1$.


  [1]: https://mathoverflow.net/questions/22801/self-homomorphisms-of-surface-groups