EDIT : I wanted to add details after the OP's comment, and I realized I forgot one case (the dihedral case $D_{2n}$ with $n$ even). Below is a slightly corrected version.

You can determine the type of the representation by looking at the density of primes $p$ such that $a_p = 0$. If that density is $\geq 1/2$, then the form is dihedral, and the converse is true. In the case of density between $1/2$ and $5/8$, the set of $p$ contains the set of primes that are inert in a quadratic field $K$, and this means your form is CM or RM by $K$ (depending of whether $K$ is imaginary or real).
In the rare cases the density is $>5/8$, it is actually $3/4$, and the projective image of the Galois representation $\rho$
is the group $\mathbb Z/2 \mathbb Z\times \mathbb Z/2\mathbb Z$, which is dihedral in three possible ways, and accordingly your form has CM by two imaginary fields and RM by a quadratic field (all three subfields of a bi-quadratic extension of the rationals).

If the density is less than $1/2$ it is either $3/8$ or $1/4$, and your in one of the so-called exceptional cases. If $\rho$ is of type $A_4$, $S_4$, $A_5$ instead, the density will be $1/4$, $3/8$ and $1/4$, so you can use this to recognize the forms that have a $\rho$ octahedral or icosahedral from the ones that have a $\rho$ tetrahedral.

How do you determine that density? It depends of what you want. If you're happy with guessing what type is your form, just count the coefficients $a_p$ non zero up to a certain $x$ with a computer and believe what it looks like. If you want something rigorous, you can use the error bound in the effective Chebotarev's density theorem, that will give you explcit values of $\epsilon(x)$ such that if the proportion of $p$ up to $x$ such that $a_p = 0$ is in an $\epsilon(x)$-range of $\geq 1/2$, $3/8$ or $1/4$, then the density is that number.

EDIT: I add some details. The proof of the stated results is easy. The representation satisfies, by Deligne-Serre, $tr \rho(Frob_p) = a_p$ for almost all $p$. The density of primes $p$ such that $a_p$ is $0$ is therefore, by Chebotarev's density theorem, the proportion of elements of trace $0$ in the subgroup $G:=$ Im $\rho$ of $GL_2(\mathbb C)$. Let $G'$ be the image of $G$ in $PGL_2(\mathbb C)$. An element $g$ of $G$ has trace $0$ if and only if $g$ has order $2$ in $G'$. We must therefore count the proportion of element of order $2$ in $G'$. For a group dihedral $D_{2n}$, this proportion is $1/2+1/(2n)$ if $n$ is even, $1/2$ if $n$ is odd. In $A_4$ there are $3$ elements of order $2$, giving a proportion of $3/12=1/4$. In $S_4$ the are
$9$ elements of order $2$, so the proportion is $9/24=3/8$. In $A_5$ there are 15 elements of order $2$ (the products of two disjoint transpositions, which form a single conjugacy class), and the proportion $15/60$ is again $1/4$. That justifies most of the assertion of the first paragraph.

To complete the second paragraph, you need a statement of Effective Chebotarev (initially due to Lagraias-Odlyzko), for which a very good reference is Serre's paper at IHES in 1981 (Quelques applications du théorème de densité de Chebotarev), available on numdam.org

below$\lambda$ precisely to avoid reaching the bound (but still have as many coefficients as possible). Perhaps I'm mistaken, or you ment something else. $\endgroup$ – Myshkin Sep 30 '15 at 15:48prime$n \le \lambda$.) $\endgroup$ – David Loeffler Sep 30 '15 at 21:11