Modular forms from counting points on algebraic varieties over a finite field Suppose we are given some polynomial with integer coefficients, which we regard as carving out an affine variety $E$, for example:
$$ 3x^2y - 12 x^3y^5 + 27y^9 - 2 = 0 \tag{$*$} $$
(We might consider a bunch of equations, we might work over projective space, but let's keep it simple for now).
We are interested in the number of points on $E$ when we reduce modulo $p$, i.e. over the finite field $\mathbb{F}_p$ as the prime $p$ varies. For our single equation in two variables, as a rough approximation, we would expect $p$ points in general. So for each prime $p$, we define numbers $a_p$ which measure the deviation from this, 
$$a_p := p - \text{number of solutions to $(*)$ over $\mathbb{F}_p$}$$
Question: When is it true that the numbers $\{a_p\}$ are "modular", in the  sense that there exists a modular form 
$$f = \sum_{n=1}^\infty b_n q^n$$
such that $b_p = a_p$ for almost all primes $p$? (The "almost all" is to avoid problems with bad primes. Note that $f$ is still uniquely determined by the above requirement.)
The Modularity Theorem of Breuil, Conrad, Taylor and Diamond says that this is true when $E$ is an elliptic curve, i.e. takes the form $y^2 = 4x^3 - g_2 x - g_3$ for some integers $g_2, g_3$. In that case, $f$ is a weight 2 modular form of level $N$ where $N$ is the "conductor" of $E$. 
But is it true for more general varieties? 
(Note: I am aware of a generalized "Modularity Theorem" for certain Abelian varieties, but it's not clear to me that what people mean by "Modular" in that context is the same as the simple-minded notion I'm using --- that an adjusted count of points mod $p$ gives the Fourier coefficients of a modular form.) 
 A: The correct setting for this construction turns out to be projective varieties, so let me suppose we have a smooth variety $X$ inside $\mathbf{P}^N$, for some $N \ge 1$, defined by the vanishing of some homogenous polynomials $F_1, \dots, F_r$ in variables $x_0, \dots, x_N$, with the $F_i$ having coefficients in $\mathbf{Q}$. Actually, let me assume the $F_i$ have coefficients in $\mathbf{Z}$, which is no loss since we can just multiply up. Then we can make sense of the reduction of $X$ modulo $p$; and we want to study the point counts $\#X(\mathbf{F}_p)$ as a function of $p$, possibly neglecting some finite set $\Sigma$ containing all primes such that the reduction of $X$ mod $p$ is singular.
Thanks to Grothendieck, Deligne, and others, we have a very powerful bunch of tools for analysing this problem. The setup is as follows. Choose your favourite prime $\ell$. Then the theory of etale cohomology attaches to $X$ a bunch of finite-dimensional $\mathbf{Q}_\ell$-vector spaces $H^i_{\mathrm{et}}(X_{\overline{\mathbf{Q}}}, \mathbf{Q}_\ell)$ (let me abbreviate this by $H^i_\ell(X)$ to save typing). The dimension of $H^i_\ell$ is the same as the $i$-th Betti number of the manifold $X(\mathbb{C})$; but they encode much more data, because each $H^i_\ell(X)$ is a representation of the Galois group $\operatorname{Gal}(\overline{\mathbf{Q}} / \mathbf{Q})$, unramified outside $\Sigma \cup \{\ell\}$; so for every prime not in this set, and every $i$, we have a number $t_i(p)$, the trace of Frobenius at $p$ on $H^i_\ell$, which turns out to be independent of $\ell$.
Theorem: $\#X(\mathbf{F}_p)$ is the alternating sum $\sum_{i=0}^{2 dim(X)} (-1)^i t_i(p)$.
Now let's analyse $H^i_\ell$ as a Galois representation. Representations of Galois groups needn't be direct sums of irreducibles, but we can replace $H^i_\ell$ by its semisimplification, which does have this property and has the same trace as the original $H^i_\ell$. This semisimplification will look like $V_{i, 1} + \dots + V_{i, r_i}$ where $V_{i, j}$ are irreducible; and the $V_{i, j}$ all have motivic weight $i$, so the same $V$ can't appear for two different $i$'s. So we get a slightly finer decomposition
$\#X(\mathbf{F}_p) = \sum_{i=0}^{2 \mathrm{dim} X} (-1)^i \sum_{j=1}^{k_i} t_{i, j}(p)$
where $t_{i,j}(p)$ is the trace of $Frob_p$ on $V_{i,j}$.
Let me distinguish now between several different types of irreducible pieces:


*

*$V_{i, j}$ is a one-dimensional representation. Then $i$ must be even, and the trace of Frobenius on $V_{i, j}$ is $p^{i/2} \chi(p)$ where $\chi$ is a Dirichlet character. 

*$V_{i, j}$ is two-dimensional and comes from a modular form. Then $t_{i,j}(p) = a_p(f)$, and $f$ must have weight $i+1$.

*$V_{i,j}$ is two-dimensional and doesn't come from a modular form. This can happen, but it's rare, and it's expected that all examples come from another kind of analytic object called a Maass wave form; in particular this forces $i$ to be even.

*$V_{i, j}$ has dimension $> 2$. Then $V_{i, j}$ cannot be the Galois representation coming from a modular form, because these always have dimension 2.


You seem to want your varieties to have $X(\mathbf{F}_p)$ = (polynomial in $p$) + (coefficient of a modular form). From the above formulae, it's clear that this can only happen if all the $V_{i,j}$ have dimension 1 or 2; there is exactly one with dimension 2 and it comes from a modular form; and the one-dimensional pieces all come from the trivial Dirichlet character. This always happens for genus 1 curves, because the $H^0$ and $H^2$ are always 1-dimensional for a curve, and the genus condition forces the $H^1$ to be two-dimensional. 
However, once you step away from genus 1 curves, this is totally not the generic behaviour, and it will only occur for unusual and highly symmetric examples, such as the rigid Calabi-Yaus and extremal $K_3$ surfaces in the links you've posted. 
A: It seems that this holds as long as one counts with respect to the appropriate "baseline". In this paper they give a class of examples. Let $X$ be a nice enough rigid Calabi-Yau threefold over $\mathbb{Q}$, with Hodge numbers $h^{i,j}$. Then, if I understand them correctly (pg 7), they say that if we set
$a_p := 1 + p^3 + (1+p)p h^{1,1} - \#X(\mathbb{F}_p)$
then these are the Fourier coefficients (for almost all primes $p$) of a modular form of weight 4 and a certain level $N$. 
In general, it seems one needs to set the $a_p$ to be the trace of the Frobenius acting on etale cohomology in some degree $i$,
$a_p = \mbox{Tr}(\mbox{Fr}_p : H^i(X) \rightarrow H^i(X)$).
In nice enough cases (such as the one above), one can interpret this trace concretely as a count of the points on $X$ mod p, with respect to an appropriate baseline.
