Monic polynomial with integer coefficients with roots on unit circle, not roots root of unity?

There are certainly non-monic polynomials of degree 4 with all roots on the unit circle, but no roots are roots of unity; $5 - 6 x^2 + 5 x^4$ for example.

Now, for a monic polynomial of degree $n$, this is impossible (I think).

So, my question is, given a monic polynomial with integer coefficients of degree $n$, what is the maximal number of roots that can lie on the unit circle, and not be roots of unity?

For example, $1 + 3 x + 3 x^2 + 3 x^3 + x^4$ has two roots on the unit circle, and two real roots.

-
A question and a remark: do you want the polynomial to be irreducible? Here is a recent paper around this problem ramanujan.math.trinity.edu/rdaileda/research/papers/p1.pdf that is investing the existence of such elements (I assume one could read out something on your question from it, but I did not really try, also but not only in view of my question for clarification). In any case perhaps the paper and the references is a starting point. –  quid Nov 28 '12 at 9:40
I think the answer is $n-2$ (at least for $n$ even). See en.wikipedia.org/wiki/Salem_number –  François Brunault Nov 28 '12 at 9:55
Just as a note: The result that if all roots are on the unit circle then they must be roots of unity goes back to Kronecker (for references and a couple of quick proofs, see mathoverflow.net/questions/10911/… ). –  Kevin P. Costello Nov 28 '12 at 18:46

There exist irreducible monic polynomials such that all their roots apart from two lie on the unique circle (and are not roots of unity). Such polynomials can be chosen among Salem polynomials and they exit in arbitrary high degree. By definition a Salem polynomial $S(x)\in \mathbb Z[x]$ is a monic irreducible reciprocal polynomial with exactly two roots off the unit circle, both real and positive. Of course non of the roots of these polynomials are roots of unity, since these polynomials are irreducible.

See for example theorem 1.6 in the article

Automorphisms of even unimodular lattices and unramified Salem numbers of Gross and Mcmullen:

http://www.math.harvard.edu/~ctm/home/text/papers/unim/unim.pdf

Theorem. For any odd integer $n\ge 3$ there exist infinitely many unramified Salem polynomials of degree $2n$.

-
Aha! That is nice! Thank you very much! –  Per Alexandersson Nov 28 '12 at 11:00

For a class of concrete examples with at least asymptotically more than $n/2$ zeros on the unit circle, the Fekete polynomials, which were just mentioned recently by Franz Lemmermeyer at this question on class numbers, might be fruitful in this regard.

Defining them by $$f_p(x)=\sum_{a=0}^{p-1}\left(\frac{a}{p}\right)x^a,$$

it seems that

$$\frac{f_p(x)}{x(x-1)}$$ when $p\equiv 3 \bmod 4$

and $$\frac{f_p(x)}{x(x-1)^2(x+1)}$$

when $p \equiv 1 \bmod 4$ are thought to be irreducible. (See another question here.) (I'd appreciate being corrected if I'm wrong on this.)

Now it has been proven in

B. Conrey, A. Granville, B. Poonen, K. Soundararajan, Zeros of Fekete Polynomials

that asymptotically more than half of the zeros lie on the unit circle.

-

Just a couple of minor top-ups to Dmitri's nice answer.

1. For each even $n\ge2$ the polynomial $p_n(x)=x^n-x^{n-1}-\dots-x+1$ is a Salem polynomial.
2. It is not known whether for any $\delta>0$ there exists a Salem polynomial such that $\theta<1+\delta$ for its largest root $\theta$ (which is a Salem number). Lehmer's conjecture suggests that the answer is no.
-

First, you are right: if a monic polynomial's roots are all on the unit circle, then every root is a root of unity. This follows from a result of Kronecker's (see: Zwei Sätze über Gleichungen mit ganzzahligen Coefficienten).

Secondly, what are you looking for is Salem Polynomials. See, for example, Definition 2.1 here. There can be arbitrarily many roots on the unit circle (none of which is a root of unity) with just two other roots located off of $S^1$.

-