We know, thanks to A. Weil, that such a function is rational, and the numerator has all of its zeros on the circle $z \overline{z} = q,$ where $q$ is the order of the field. The question is: can every polynomial in $\mathbb{Z}[x]$ with the zeros on the circle arise in this way? Is there some obvious and/or conjectural obstruction?

6$\begingroup$ This is a duplicate of a question I asked here: mathoverflow.net/questions/70605/fromzetafunctionstocurves. $\endgroup$ – Peter Humphries Aug 10 '14 at 17:48

$\begingroup$ @PeterHumphries Oops, did not see your question... $\endgroup$ – Igor Rivin Aug 10 '14 at 20:22
Tate and Honda show that almost all polynomials like that are the Lfunction of an abelian variety over the finite field. The problem with curves is much harder and it's open (for genus g>2). One necessary condition is that the expression giving the number of points on the curve ($q+1$ minus sum of zeros) has to be positive. This is not true for every polynomial with your conditions, so the answer to your first question is no. That's an obstruction but not the only one.
For genus 2, see http://arxiv.org/abs/math/0607515

$\begingroup$ Thanks! Another question (which is vacuous in genus 2) is whether my question is any easier in the hyperelliptic case?! $\endgroup$ – Igor Rivin Aug 10 '14 at 20:20

1$\begingroup$ @IgorRivin Probably not. There are more constraints (number of points at most $2q+2$, etc) but I don't think there is an answer yet. $\endgroup$ – Felipe Voloch Aug 10 '14 at 20:56

$\begingroup$ Even for hyperelliptic curves, we aren't anywhere close to having a conjecture of which zeta functions actually occur. But the asymptotic distribution of these zeta functions is known when $q\gg g$ (KatzSarnak) and when $g\gg q$ (KurlbergRudnick and subsequent authors). $\endgroup$ – Michael Zieve Aug 11 '14 at 3:08
When $q$ is much larger than $g$, and $g$ is at least $10$ or so, one expects that very few of the polynomials satisfying the constraints in the question will actually occur as a zeta function. The reason is that every abelian variety has a zeta function resembling the zeta function of a Jacobian variety, which in turn contains the same information as the zeta function of the corresponding curve. In particular the "$H^1$part" of the zeta function of a $g$dimensional abelian variety satisfies the stated constraints (see e.g. Milne's articles in the CornellSilverman book for this). But the number of $g$dimensional principally polarized abelian varieties is roughly $q^{(g^2+g)/2}$, while the number of $g$dimensional Jacobian varieties is roughly $q^{3g3}$ (assuming $g>1$). So, unless $g$ is tiny, a vanishingly small fraction of $g$dimensional PPAV's over $\mathbf{F}_q$ are actually Jacobians. This doesn't tell the full story since the zeta function of an abelian variety is determined by the isogeny class of the variety, rather than the isomorphism class, so the real question is which isogeny classes of PPAV's contain Jacobians. But there are heuristics about sizes of isogeny classes, which suggest that the vast majority of such classes do not contain Jacobians.
All of this assumed that $q$ is much bigger than $g$, so that (for instance) the $(3g3)$dimensional moduli space of curves will have approximately $q^{3g3}$ points over $\mathbf{F}_q$. If $q$ is much smaller than $g$ then one does not know a decent estimate for the number of genus$g$ curves over $\mathbf{F}_q$, since when applying Deligne or LangWeil to the moduli space one gets a formula for the number of $\mathbf{F}_q$points (in the moduli space) in which the error term is much bigger than the main term. The situation with $q$ smaller than $g$ is where the constraint Felipe mentioned plays a big role, since many polynomials satisfying the constraints in the question would correspond to curves having a negative number of $\mathbf{F}_q$rational points if they occurred as a zeta function.

$\begingroup$ Just a remark: the related question of whether every AV over a finite field is (geometrically) isogenous to a Jacobian is very interesting and, as far as I know, very open. $\endgroup$ – Daniel Litt Aug 11 '14 at 6:21
Here is another obstruction (probably the one hinted to by Felipe Voloch). There are asymptotic bounds for the number of points of a curve of (large) genus $g$ over a given field of cardinality $q$. For example, Drinfeld and Vladut proved that $\# C(\mathbf F_q)\leq (\sqrt q1+\mathrm o (g))g$, when $g\to\infty$.
However, one can presumably construct sequences of monic polynomials $(P_g)$ in $\mathbf Z[T]$, with $\deg(P_g)=2g$, whose roots are $q$Weil numbers, and such that $$ \liminf(q+1s(P_g))/g=\limsup s(P_g)/g>\sqrt q1 $$ when $g\to\infty$. (I have denoted by $s(P)$ the sum of the roots of a polynomial $P$.) Then only finitely many such polynomials could appear for a curve over $\mathbf F_q$.

1$\begingroup$ Yes, that is the same constraint Felipe mentioned: the DrinfeldVladut result follows from the fact that $\#C(\mathbf{F}_q)\le\#C(\mathbf{F}_{q^n})$ for all $n$. An example of nonoccurring polynomials is $P_g(T)=(T\sqrt{q})^{2g}$ when $q$ is a square and $g>(q+1)/(2\sqrt{q})$, as a corresponding curve over $\mathbf{F}_q$ would have $q+12g\sqrt{q}$ rational points, which is negative. It is easy to construct many analogous examples when $g$ is large compared to $q$. $\endgroup$ – Michael Zieve Oct 26 '14 at 5:42