Why does the definition of modularity demand weight 2? Allow me to quote a definition from Gelbart in "Modular Forms and Fermat's Last Theorem":
Definition. Let $E/\mathbb{Q}$ be an elliptic curve. We say that $E$ is modular if there is some normalised eigenform
$$ f(z) = \sum_{i=1}^{\infty} \ a_ne^{2\pi inz} \in S_2(\Gamma_0(N),\epsilon), $$
for some level $N$ and Nebentypus $\epsilon$, such that
$$ a_q = q + 1 - \#(E(\mathbb{F}_q)) $$
for almost all primes $q$.
This is the basic question of the post:

Why is the weight of $f$ taken to be 2? Can I instead take 3, or 4, or 5, or even 19/2, without disturbing the peace?

I am aware of other definitions of modularity, some of which don't mention modular forms at all, but nonetheless I feel that weight 2 lurks beneath all of these.
I think one approach would involve differentials, and the construction of Eichler-Shimura, but I'm not so sure. Further, perhaps there are several reasons which fit together to tell a nice story.
Is it a corollary of this question that it doesn't matter what the weight is?
Finally, can I replace $E$ above with any abelian variety, and ask the same question?
 A: I started to write this as a comment to the original question, but it became too long. It is not disjoint from the previous expert responses, but it emphasizes a particular viewpoint.
One can (and one should) shift normalize any automorphic (in particular any modular) $L$-function so that the functional equation relates $s$ to $1-s$. Then the gamma factors identify the archimedean component of the underlying automorphic form much like the Euler factors identify the non-archimedean components. In particular, if the $L$-function of a holomorphic cusp form is so normalized (i.e. $s$ is related to $1-s$), then the gamma factors determine the weight (and vice versa). I say gamma factors, because the usual single gamma factor can be factored into two gamma factors by the doubling formula for the gamma function (for a modular $L$-function the "true gamma factors" form a pair).
So you can ask your question as follows. If we shift normalize the $L$-function of an elliptic curve so that the functional equation relates $s$ to $1-s$, then why are the gamma factors always the same? In order to ask this question you need to assume already that the $L$-function obeys a rather specific functional equation with gamma factors, and then the question inquires what the gamma factors can be. 
It is possible that the fairest answer to this question (i.e. your question) is as follows:
It was an experimental fact that the gamma factors are always the same, hence the precise form of the modularity conjecture was formulated, which then turned out to be right, namely it was proved by great efforts of great mathematicians.
Once again, normalization of all automorphic $L$-functions is key in this discussion, it should not be underestimated. It is not the usual normalization favored by algebraic people.
Added: In a leisurely style one could say the following. The first miracle is that the $L$-function of an elliptic curve is entire and satisfies some functional equation. The second miracle is that the $L$-function is automorphic, as suggested by the functional equation. More specifically, it looks like a $\mathrm{GL}_2$ automorphic $L$-function. Not only it is $\mathrm{GL}_2$, but it comes from a very specific modular form, namely a holomorphic form, let's call this the third miracle. Then, as a final miracle, this holomorphic form is always of weight 2 whose level can also be specified in terms of the elliptic curve.
A: In terms of the data of which you speak, you can use asymptotics of the Fourier coefficients on the modular form side and the Hasse bound on the elliptic curve side. Specifically, if $f$ is a normalized newform of weight $k$ (an integer $\geq2$, I'm not enough of an expert on this to know if more general $k$ are allowed), then
$$\sum_{n\leq X}|a_n|^2/n^{k-1}=c_fX+O(X^{3/5})$$
(see section 14.9 of Iwaniec–Kowalski), but, on the other hand, the Hasse bound for an elliptic curve states that
$$|a_p|<2\sqrt{p}$$
(see this mathoverflow answer to find out what this implies about the $a_n$ of the putative cusp form attached to the elliptic curve). Given this bound, $k$ must be at most 2.
Update: From reading the mathscinet review of an old paper of Selberg's, it looks like the asymptotic on the Fourier coefficients (I guess originally due to Rankin) works for any weight which is a positive real number (but the language in the review is rather old-fashioned and I don't have access to the paper right now). I'll take a look probably in a couple of days.
A: $\newcommand\Q{\mathbf{Q}}$
$\newcommand\Qbar{\overline{\Q}}$
$\newcommand\Gal{\mathrm{Gal}}$
$\newcommand\C{\mathbf{C}}$
$\newcommand\Sym{\mathrm{Sym}}$
$\newcommand\E{\mathcal{E}}$
$\newcommand\Betti{\mathrm{Betti}}$
$\newcommand\Z{\mathbf{Z}}$
$\newcommand\Hom{\mathrm{Hom}}$
$\newcommand\T{\mathbf{T}}$
To answer this question, it might be best to start with 
the following:
Q. What do the Galois representations attached to a variety know about the variety?
In order make this more precise, let us introduce some notation. Fix a prime $p$ and a non-negative
integer $n$. 
Let $X$ be a proper smooth scheme over $\Q$, and
let $V = H^n_{et}(X/\Qbar,\Q_p)$ denote the $n$th etale cohomology group
of $X$. The basic and fundamental properties of etale cohomology tell us that:


*

*$V$ is a vector space of dimension $H^n_{\Betti}(X(\C))$, where $H_{\Betti}$
denotes Betti (or singular) cohomology, and $X(\C)$ denotes the complex points of $X$ thought
of as a topological manifold.

*$V$ (with the $p$-adic topology) has a continuous action of $G_{\Q}:=\Gal(\Qbar/\Q)$.
Grothendieck and Serre further conjecture that the $G_{\Q}$-representation $V$
is semi-simple.
The strongest possible conjecture one might make is to ask whether the functor from smooth projective varieties over $\Q$ to
semi-simple $G_{\Q}$-representations (or the collection of all such representations for $n \le 2 \cdot \mathrm{dim}(X)$)
is fully faithful. However, this is too much to ask, for the following reasons.
(i). The target category is semi-simple, but the category of varieties is far from
semi-simple. (In particular, the existence of a map
$X \rightarrow Y$ does not imply the existence of a non-trivial map $Y \rightarrow X$.)
(ii). Varieties built in a combinatorial way from projective spaces (think toric
varieties) tend to have etale cohomology groups indistinguishable from 
products of projective spaces. This is because their cohomology groups are generated by geometric cycles, on which Galois
acts in a well understood way (essentially by some power of the cyclotomic character).
These are - in some sense - manifestations of the same reason: A correspondence 
in $X \times Y$ gives rise to a cohomology class in $H^*(X \times Y)$; then by
the Künneth formula, this leads to a relation between the cohomology
of $X$ and $Y$ even when there is not necessarily any non-trivial
map from $X$ to $Y$ (or vice versa).
In order to account for this, one can try to take the quotient category of the
category of algebraic varieties in which one is allowed to "break up" smooth
proper varieties into pieces given the existence of certain correspondences on $X$. There are a variety of ways in which one might do this. Conjecturally, these constructions are all essentially the same, and the corresponding category is the category of  pure motives.  The Tate conjecture now says that etale cohomology is a fully faithful functor from pure motives to semi-simple $G_{\Q}$-representations.
 Example  If $E$ is an elliptic curve over $\Q$, and $n = 1$, then 
the etale cohomology group $V$  is the (dual) of the usual representation
attached to the $p$-adic Tate module of $E$. Suppose that $E'$ is another
elliptic curve over $\Q$ with first etale cohomology group $V'$. 
For curves, the theory of "motives" is essentially the theory of abelian varieties. (More generally, the theory of $H^1$ is essentially the theory of abelian
varieties, since, for any proper variety $X$, there is an isomorphism $H^1(X) \simeq H^1(A(X))$, where $A(X)$ is the Albanese of $X$.)
Tate's conjecture in this case says that
$$\Hom(E,E') \otimes \Q_p \rightarrow \Hom_{G_{\Q}}(V,V')$$
is an isomorphism. This is how you will see the Tate conjecture stated for elliptic
curves, for example, in AOEC. The Tate conjecture for abelian varieties is a theorem of Faltings.
(Suggestion: to understand what the Tate conjecture really is about, and why it is hard, you should really think about the special case of Elliptic curves.)
If we now return to our question, we can (tautologically) say the following: assuming the Tate conjecture, the etale cohomology knows about the motive corresponding to the original variety. What does that really mean? One way of thinking about motives is as a ``universal cohomology theory''. In particular, we can recover from the motive not only the etale cohomology groups, but also the algebraic
de Rham cohomology groups. Recall that de Rham cohomology is another cohomology theory
that gives vector spaces of the "correct" dimension for a smooth proper variety $X/\Q$. The de Rham cohomology groups do not have associated Galois representations, but
they  do  have a Hodge filtration. Over $\C$, if one takes the associated graded of the Hodge filtration, one recovers the Hodge decomposition:
$$H^n_{dR}(X,\C) = \bigoplus_{p+q=n} H^{p}(\Omega^q_X).$$
The dimensions of the latter space are called the Hodge numbers $h^{pq}$. 
So, assuming the Tate conjecture, from $V$ we can  recover the underlying 
motive, from which we may reconstruct the de Rham cohomology, and then the Hodge numbers. The Tate conjecture  seems to be very hard.
However,  Grothendieck asked the following: given
$V$, can we directly recover the (algebraic $p$-adic) de Rham cohomology along with its filtration  without 
first constructing the motive?  This was a great question, and the answer (yes!) constitutes one of the major achievements of $p$-adic Hodge Theory. I can do no more than give a cartoon description here. In order to do so, first recall the much more classical story connecting de Rham cohomology to Betti  (singular) cohomology. These groups can both naturally be defined as vector spaces over $\Q$ (one has to define de Rham cohomology in the correct way), but the isomorphism relating these spaces comes from integrating  forms  over  cycles.  Yet these integrals are typically 
 transcendental numbers, so to pass from Betti to de Rham cohomology one first has to tensor with a field
bigger than $\Q$ which contains all these periods (usually, one simply tensors with $\C$).
 In order to pass from etale cohomology to algebraic de Rham cohomology, one might ask for a period ring in which we can compare both groups. In this refined setting, the period ring should both have a Galois action and a filtration. The most basic verion of a period ring is $B_{HT}$, specifically,
$$B_{HT} := \bigoplus_{\Z} \C_p(n),$$
where $\C_p$ is the completion of $\Qbar_p$, and $\C_p(n)$  is $\C_p$ twisted
(as a local Galois module) by the $n$th power of the cyclotomic character.
 The ring
$B_{HT}$ has a natural filtration (indeed, it is even graded). Now we can consider 
$$D_{HT}(V) = (V \otimes B_{HT})^{\Gal(\Qbar_p/\Q_p)}.$$
The Galois group acts  on  both  $V$ and $B_{HT}$. The result is a graded
(and so filtered) module. On the other hand, one can also consider the ring
$B_{HT} \otimes H^n_{dR}(X/\Q_p)$, where there is a natural way to make sense of the
corresponding filtration. An important theorem of Faltings then says that
$$H^{n}(X/\Qbar_p,\Q_p) \otimes B_{HT} = H^n_{dR}(X/\Q_p) \otimes B_{HT},$$
and $D_{HT}(V) = H^n_{dR}(X/\Q_p)$. In particular, from a geometric Galois
representation, we can recover the Hodge filtration and the Hodge numbers.
 Modular Forms.  The Eichler-Shimura isomophism relates modular forms of weight $k \ge 2$
 to $H^1(X_0(N),\Sym^{k-2}\Q)$. If $k = 2$, this is just $H^1(X_0(N),\Q)$. The Hecke algebra
$\T$ acts on $H^1_{\Betti}(X_0(N),\Q)$, and (since it is constructed functorially) also on the etale  cohomology
$H^1(X_0(N),\Q_p)$. Now the Hodge decomposition of $H^1$ is $H^1 = H^{0,1} \oplus H^{1,0}$, where
$h^{0,1} = h^{1,0}$ is the genus of $X_0(N)$. 
The Hecke algebra breaks up the cohomology into two dimensional pieces corresponding to the Galois representations
associated to eigenforms; it turns out that each two dimensional piece contains one dimension from
$H^{0,1}$ and one dimension from $H^{1,0}$. The result of Faltings above tells us that we can read off
that $h^{0,1} = h^{1,0} = 1$ directly from the Galois representation.
For $k > 2$, recall that (technical issues aside) there is a universal elliptic curve $\E \rightarrow X_0(N)$.
The Kuga-Sato variety is (again, roughly) The $k-1$ dimensional variety $K = \E \times_X \E \ldots \times_X \E$ where
$X = X_0(N)$. There is  a natural map $\pi: K \rightarrow X$. 
The local system $\Sym^{k-2}(\Q^2_p)$ is trivialized over $K$, and so, using the proper base change theorem,
Deligne shows that  $H^1(X_0(N),\Sym^{k-2}\Q_p)$ is a sub-quotient  of the cohomology group $H^{k-1}(K,\Q_p)$.
(Warning: this requires more than simply a formal cohomological argument, it also requires some trickiness with weights to show
that terms on different diagonals the Leray spectral sequence don't "mix", and hence the sequence degenerates.) The Galois representation associated to a modular form
is now a two-dimensional piece of $H^{k-1}(K,\Q_p)$. Faltings proves that the corresponding "piece" of de Rham cohomology
seen by this representation is $H^{0,k-1} \oplus H^{k-1,0}$. In particular, the representation has Hodge numbers
$h^{0,k-1} = 1$ and $h^{k-1,0} = 1$.
Given a Galois representation $V$, one can twist $V$ by the cyclotomic character. How does this effect the Hodge decomposition?
One can compute this on the Hodge side by seeing what happens to the cohomology of $X \times \mathbf{G}^1_m$ and comparing with
the Künneth formula. It turns out that $h^{p,q}(V(n)) = h^{p-n,q-n}$. Thus, if only know $V$ up to twist, we still recover some information
about the Hodge numbers.
 Returning to modular forms.  The coefficients $a_p$ determine the Galois representation, by Cebotarev. 
A modular form of weight $k$ has Hodge numbers $h^{0,k-1} = h^{k-1,0} = 1$. The determinant of the representation is the $k-1$th power
of the cyclotomic character (up to a finite character) which can be read off from the "degree". By twisting, we can easily change the determinant,
and change the Hodge numbers to $h^{-d,k-d-1} = h^{k-d-1,-d} = 1$.  Yet,
it is clear that we cannot twist so that $h^{1,0} = h^{0,1} = 1$ unless $k = 2$. Thus, given a modular form of weight $k > 2$, it cannot be associated to
an elliptic curve even after twisting.   This is Kevin's answer.  
Secondly, any motive has (conjecturally) an $L$-function. The recipe of building this $L$-function breaks up into two parts. The first involves the factors
at finite primes, which give rise to the Euler product. The second involves the infinite primes, which give rise to Gamma factors. The information
at $\infty$, however, (by Tate's conjecture) can be read off from the Galois representation, and the recipe of Deligne shows that it will exactly depend
on the Hodge numbers of the motive, and visa versa.  Moreover, twisting by $\epsilon^k$ some power of the cyclotomic factor has the effect of replacing $L(s)$ by $L(s+k)$ (and shifting the corresponding central value) In particular, given an elliptic curve, one knows the Gamma factors (because one knows the Hodge decomposition of $E$), and one sees that even after twisting one cannot get Gamma factors that
"look like" the Gamma factors associated to a modular form of weight different from $2$.  This is GH's answer. 
More generally, arithmetic conjectures of Langlands type imply that all motives should be "automorphic", and that the Hodge structure of the motive determines the infinity type of the automorphic form,
which in turn determines the Gamma factors. So, at least morally, given a pure irreducible motive $V$, we know that  if  it is automorphic, it must be automorphic of a particular weight determined by the underlying geometry of $V$.
Of course, even before the result of Faltings, one had enough faith in terms of how these things were connected to be very confident that Elliptic curves over $\Q$ should
correspond exactly to weight two forms - GH's remark that "It was an experimental fact that the gamma factors are always the same, hence the precise form of the modularity conjecture was formulated, which then turned out to be right, namely it was proved by great efforts of great mathematicians" seems spot on.
 Related problems. 
Given a modular eigenform $f = \sum a_n q^n$ of weight four (in the arithmetic normalization), one can ask: is it possible to show that there does not exist a weight
two modular form
$g = \sum b_n q^n$ where $a_p = p b_p$ for all primes $p$   without  using $p$-adic Hodge theory? I think this is not so easy. For example:
(i) The arithmetic approach: The weight $4$ form $f$ would have the property that it is not ordinary at every prime, since clearly
$a_p = p b_p \equiv 0 \mod p$. One conjectures that a set of primes of density one are modular (or $1/2$ if $f$ has CM). Yet it is still unknown whether
any form of weight $\ge 4$ has even a single ordinary prime.
(ii) The analytic approach: What do the distributions of coefficients of weight $2$ and $4$ forms look like? Sato-Tate says that the normalized coefficients satisfy a precise 
distribution (now a theorem!). Yet the "normalized" coefficients of $f$ and $g$ are by construction exactly the same, so Sato-Tate says nothing. In particular,
it is hard to see any analytic estimates of functions involving the $a_p$ being able to distinguish two classes of numbers with the same
underlying distribution.  A related argument: The Hasse
bound in weight four is satisfied by $p b_p$ if and only if the weight two Hasse bound is satisfied by $b_p$.
 Summary.   Conjectures arise organically from heuristics and computation. I was "known" that Elliptic curves should be associated to weight two forms long before
one could actually formally prove that they  weren't  associated to twists of weight $4$ forms. To prove the latter fact, one has to use $p$-adic Hodge theory.
(* I am not sure about a lot here, so I'm putting this community wiki. Also, there is no mention of weight 1 or half integers. Change whatever needs changing, or in the extreme cases, peacefully leave a comment to delete...)
A: There has been a lot written already about this question. but here is a simple answer. The Hodge--Tate weights of the Tate module of an elliptic curve are 0 and 1. The Hodge--Tate weights of the Galois representation associated to a weight $k$ modular form are 0 and $k-1$. So if the Tate module is associated to a modular form, it's associated to a weight 2 modular form.
A: The reason why the weight needs to be $k=2$ becomes clear when you consider this (equivalent) version of modularity: $E/\mathbb{Q}$ is modular if there there exists a normalized newform $f$ for $\Gamma_0(N)$ such that $L(f,s) = L(E,s)$, i.e., the $L$-function attached to $f$ coincides with the Hasse-Weil $L$-function attached to $E/\mathbb{Q}$.
Now, the $L$-function of a normalized newform of weight $k$ for $\Gamma_0(N)$ has an Euler product of the form:
$$L(f,s) = \prod_{p|N} \frac{1}{1-\lambda_p p^{-s}} \prod_{p\nmid N} \frac{1}{1-\lambda_p p^{-s} + p^{k-1-2s}}.$$
The $L$-function of $E$ is defined as an Euler product:
$$ L(E,s) = \prod_{p\geq 2}\frac{1}{L_p(p^{-s})}, $$
where $L_p(T) = 1-a_pT+pT^2$ if $E$ has good reduction at $p$, $L_p(T)=  1-T$ if $E$ has split multiplicative reduction at $p$, $L_p(T) = 1+T$ if $E$ has non-split multiplicative reduction at $p$
 and $L_p(T) = 1$ if $E$ has additive reduction at $p$.
In particular, $L_p(T)=1-a_pT+pT^2$ for almost all primes (for all primes $p\nmid N(E)$). If there is any hope that these two Euler products agree, then we must have $k=2$.
