This is essentially a string of comments thatwould that would not fit into the box, with one answer in the middle.
@Kevin: for instance, we could take $X$ to be a variety over a $p$-adic field $K$ with good reduction. I agree that the question should begin by making this explicit. The Betti numbers have an interpretation in terms of $\ell$-adic cohomology, so are independent of the choice of embedding $K \hookrightarrow \mathbb{C}$.
@OP: Note that the zeta function cannot be a birational invariant because the Betti numbers are not birational invariants: if you blow up e.g. a surface at a point, $B_2$ increases by $1$. The fact that, as Felipe says, the zeta function is a rational isogeny invariant of abelian varieties -- along with the implication that, for algebraic curves, the zeta function depends only on the isogeny class of the Jacobian -- is the strongest invariance statement I know of along these lines.
The question about Calabi-Yau's seems interesting, although one should say exactly what one means by a Calabi-Yau variety; there is more than one definition.
The question about the Frobenius eigenvalues seems prohibitively vague to me: I do not know what it means to "explain" them.
