Allow me to quote a definition from Gelbart in "Modular Forms and Fermat's Last Theorem":
Definition. Let $E/\mathbb{Q}$ be an elliptic curve. We say that $E$ is modular if there is some normalised eigenform
$$ f(z) = \sum_{i=1}^{\infty} \ a_ne^{2\pi inz} \in S_2(\Gamma_0(N),\epsilon), $$
for some level $N$ and Nebentypus $\epsilon$, such that
$$ a_q = q + 1 - \\#(E(\mathbb{F}_q)) $$$$ a_q = q + 1 - \#(E(\mathbb{F}_q)) $$
for almost all primes $q$.
This is the basic question of the post:
Why is the weight of $f$ taken to be 2? Can I instead take 3, or 4, or 5, or even 19/2, without disturbing the peace?
I am aware of other definitions of modularity, some of which don't mention modular forms at all, but nonetheless I feel that weight 2 lurks beneath all of these.
I think one approach would involve differentials, and the construction of Eichler-Shimura, but I'm not so sure. Further, perhaps there are several reasons which fit together to tell a nice story.
Is it a corollary of this question that it doesn't matter what the weight is?
Finally, can I replace $E$ above with any abelian variety, and ask the same question?