Having resolved my ignorance concerning surface groups I can now answer question 2 negatively. It is impossible if $Y$ is (for example) an oriented surface of genus at least $2$.
Suppose that $f: Y \to Y$ is a self map of the surface such that the eigenvalues of $f^*$ acting on each $H^i(Y)$ are all nonzero (otherwise we can't "detect" the betti numbers), and such that $H^i(Y)$ and $H^j(Y)$ do not have eigenvalues of common magnitude for $i \neq j$. Then in particular $f^*$ acts on $H^2(Y)$ nontrivially, say by multiplication by some integer $d$. This integer cannot be $\pm 1$ since then $H^0(Y)$ and $H^2(Y)$ would contain eigenvectors with eigenvalues of equal magnitude.
Consider the subgroup $H = f_*(\pi_1(Y))$ inside $G = \pi_1(Y)$. If this had infinite index, then $f$ would lift to a map to some infinite covering of $Y$, hence would induce a trivial map of $H^2$. So $H$ has finite index in $G$. Let $X \to Y$ be the corresponding covering space. Then the fundamental group of $X$ is a quotient of $\pi_1(Y)$, hence its abelianization has rank $\leq 2g$ where $g$ is the genus of $Y$. This implies that $X$ is a closed surface of genus at most $g$. But its Euler characteristic is precisely $[G:H]$ times the Euler characteristic of $Y$, hence $X = Y$. Thus $f$ induces a surjection on $\pi_1(Y)$. By the post cited above, $f$ actually induces an isomorphism on $\pi_1(Y)$, hence it is a homotopy equivalence. In particular, $d = \pm 1$, contrary to assumption.