Characters of weight 1 cusp forms Assume that the space of cusp forms of weight 1 $S_1(\Gamma_0(N),\chi)$ is
non zero. What can one say about the odd character $\chi$, for instance
concerning its order ? Serre tells me that a theorem of Tate implies that
$\chi$ is of order $2$ or $4$ if $N$ is prime, and that this can probably
be generalized to $N$ squarefree, and more difficult if not. What is known,
or conjectured ?
 A: $\def\Q{\mathbf{Q}}$
$\def\Z{\mathbf{Z}}$
$\def\Gal{\mathrm{Gal}}$
$\def\GL{\mathrm{GL}}$
$\def\Qbar{\overline{\mathbf{Q}}}$
$\def\C{\mathbf{C}}$
$\def\OL{\mathcal{O}}$
$\def\PGL{\mathrm{PGL}}$
$\def\rhobar{\overline{\rho}}$
Dear Henri, I think the case of prime conductor is a little special. The order of $\chi$ can be quite big in the dihedral case even when $N$ is a product of two primes.
First, here is a reminder of how the conductor works when $p$ exactly divides $N$. If $f$ is a classical modular cusp form in $S_1(\Gamma_0(N),\chi)$, then the representation $\rho$ associated to $f$ has the property that, if $p \| N$,
$$\rho |_{I_p} \simeq \left( \begin{matrix} \chi_p & 0 \\
0 & 1 \end{matrix} \right),$$
where $I_p \subset D_p = \Gal(\Qbar_p/\Q_p)$ is the inertia and decomposition group at $p$. This is because (when the image of inertia is tame) the conductor is given by $2 - \mathrm{dim} V^{I_p}$. As a consequence, we see that the image of $I_p$ under $\rho$ is isomorphic to the image of $I_p$ under the projective representation when $p \| N$.
Lemma: If $N$ is squarefree and $\rho$ has exceptional projective image, then $\chi^{60}$ is trivial.
Proof: The projective image of $\chi_p$ is a cyclic subgroup of $A_4$, $S_4$, or $A_5$, and hence has order $\le 5$. Hence $\chi^{60}_p$ is unramified at $p$. It follows that $\chi^{60}$ is unramified at all primes, and is hence trivial because $\Q$ has no unramified abelian extensions.
Remark: If there is only one prime, then the same argument shows that $\chi$ has order at most $5$. Since $\chi$ is odd, it has even order, and so order $2$ or $4$. 
Remark: One can do slightly better: If the exceptional subgroup is contained in $S_4$ then one gets $12$ instead of $60$, and since $A_5$ has no elements of order $4$, one gets $30$ in that case (or $6$ in the $A_4$ case). 
Example:Let $K$ be the splitting field of $x^5 - 33x^3 - 55x^2 + 803x - 1199$. The Galois group $\Gal(K/\Q)$ is $A_5$, and $K$ is odd. There is a corresponding projective representation:
$$\rhobar: G_{\Q} \rightarrow \Gal(K/\Q) = A_5 \rightarrow \PGL_2(\mathbf{C})$$
The field
 $K$ is ramified at $7$, $11$, and $197$ only (and $\infty$), and
$$I_{7} = D_{7} \simeq \Z/3\Z,$$
$$I_{11} = D_{11} \simeq \Z/5\Z,$$
$$I_{197} = \Z/2\Z, \qquad D_{197} = \Z/2\Z \oplus \Z/2\Z.$$
Each of these decomposition groups inside $\PGL_2(\C)$ admit lifts $D_p \rightarrow \GL_2(\C)$ on which inertia has non-trivial invariants. (This would be false if $D_p$ was dihedral of order $6$ or $10$ and inertia was the index two normal subgroup, or alternatively if $I_p = D_p$ was the Klein $4$-group.) Recall
the theorem of Tate: the projective representation
$$\rhobar: G_{\Q} \rightarrow \Gal(K/\Q) = A_5 \rightarrow \PGL_2(\mathbf{C})$$
admits a lift $\rho: G_{\Q} \rightarrow \GL_2(\C)$ with the following property: for each prime $\ell$,
if we choose a local lift $\rho_{\ell}: D_{\ell} \rightarrow \GL_2(\C)$ of $\rhobar_{\ell}:=\rhobar |D_{\ell} \rightarrow \PGL_2(\C)$, then we may choose $\rho$ to have the property that it agrees with all these lifts on inertia, that is, $\rho |I_{\ell} = \rho_{\ell}| I_{\ell}$. Hence, in our particular example, we may ensure that $\rho$ is
 unramified outside $7$, $11$, $197$, and moreover, for each of these primes $p$, the image of inertia has a fixed line. By the Artin conjecture (known in this case by Buzzard-Taylor), it follows that $\rho$ is modular of level $7 \cdot 11 \cdot 197$, and the Nebentypus character will have order $30$.
The Dihedral Case:
However, things can be arbitrarily bad in the dihedral case as soon as $N$ is divisible by two primes.
Fix a prime $\ell \equiv -1 \mod 4$, and let $K = \Q(\sqrt{-\ell})$.
The unit group of the ring of integers $\OL_K$ of $K$ has order $2$. Now let $p$ be an auxiliary prime which splits in $K$, and let $v$ and $w$ be the primes above $p$ in $\OL_K$. The ray class group of conductor $v$ has order $(p-1)/2$, and hence there exists a character:
$$\psi: G_K \rightarrow \C^{\times}$$
of order $(p-1)/2$ which is ramified only at $v$. Now let us consider the induction
$$\rho = \mathrm{Ind}^{\Q}_{K} \psi; \qquad \rho | G_K = \psi \oplus \psi^c$$
It is odd, because $K$ is not real. So it does correspond to a weight one modular form.  I claim it has conductor $p \ell$. It suffices to show that $\rho$, restricted to the decomposition group at either $p$ or $\ell$, is tamely ramified and has an invariant line. For $\ell$, we have $\rho | I_{\ell} = \eta \oplus 1$ for the quadratic character $\eta$ coming from $\Gal(K/\Q)$ which is tamely ramified because $\ell > 2$. For the prime $p$, we may as well restrict to the decomposition group at $w|p$. Yet $\psi$ is unramified at $w$ by construction, so $\rho | I_p = 1 \oplus \psi^c |_{I_p}$.
So the level is $N = p \ell$. Let's compute the nebentypus, which is the determinant.
Let $\chi$ be the determinant of this representation. Then
$$\chi |_{G_K} = \psi \psi^c.$$
I claim that this has order $(p-1)/2$, and that $\chi$ has order $\mathrm{lcm}(2,(p-1)/2)$. To see this, note that $\psi$ is  ramified at $v$ to this order and is unramified at $w$, whereas $\psi^c$ is ramified at $w$ to this order and is unramified at $v$. Hence $\chi$ restricted to $G_K$ is ramified at both $w$ and $v$ to order $(p-1)/2$, and so $\chi$ itself has order $p-1$. Note that $\chi^{(p-1)/2}$ is unramified at $p$ and $\chi^2$ is unramified at $\ell$, so $\chi^{\mathrm{lcm}(2,(p-1)/2)}$ is trivial.
Why is the dihedral case OK when $N = p$ is prime?
In this case, since $\Q$ has no unramified extensions, the quadratic extension $K$ must be $\Q(\sqrt{p (-1)^{(p-1)/2}})$. However, $p$ is ramified in this extension, so the only way a character can be unramified at a prime above $p$ is if it is unramified everywhere, in which case $\chi^2$ is unramified everywhere, and $\chi$ has order two. 
