Having resolved my [ignorance][1] concerning surface groups I can now answer question 2 negatively.  It is impossible if $Y$ is (for example) an oriented surface of genus at least $2$.

Suppose that $f: Y \to Y$ is a self map of the surface such that the eigenvalues of $f^*$ acting on each $H^i(Y)$ are all nonzero (otherwise we can't "detect" the betti numbers), and such that $H^i(Y)$ and $H^j(Y)$ do not have eigenvalues of common magnitude for $i \neq j$.  Then in particular $f^*$ acts on $H^2(Y)$ nontrivially, say by multiplication by some integer $d$.  This integer cannot be $\pm 1$ since then $H^0(Y)$ and $H^2(Y)$ would contain eigenvectors with eigenvalues of equal magnitude.

Consider the subgroup $H = f_*(\pi_1(Y))$ inside $G = \pi_1(Y)$.  If this had infinite index, then $f$ would lift to a map to some infinite covering of $Y$, hence would induce a trivial map of $H^2$.  So $H$ has finite index in $G$.  Let $X \to Y$ be the corresponding covering space.  Then the fundamental group of $X$ is a quotient of $\pi_1(Y)$, hence its abelianization has rank $\leq 2g$ where $g$ is the genus of $Y$.  This implies that $X$ is a closed surface of genus at most $g$.  But its Euler characteristic is precisely $[G:H]$ times the Euler characteristic of $Y$, hence $X = Y$.  Thus $f$ induces a surjection on $\pi_1(Y)$.  By the post cited above, $f$ actually induces an isomorphism on $\pi_1(Y)$, hence it is a homotopy equivalence.  In particular, $d = \pm 1$, contrary to assumption.  

  [1]: http://mathoverflow.net/questions/22801/self-homomorphisms-of-surface-groups