What is exceptional about the prime numbers 2 and 3? Admittedly this question is  vague. But I hope to convey my point. Feel free to downvote this.
Permit me to define prime number  the following way:
A number $n>1$ is a prime if all integers $d$ with $1< d \leq \sqrt{n}$ give non-zero remainders  while dividing $n$.   This will expose the primes 2 and 3: they qualify as primes  as this condition is vacuous (there are simply no integers in that interval to check divisibility).
Lot of exceptions to  some theory (quadratic forms, elliptic curves, representation theory)  happen at primes 2 and 3. Is it because 2 and 3 entered The Prime Club by this backdoor?  It is my wild guess. Perhaps experts might be able to say if I am wrong. Or possibly there are more conceptual reasons. Kindly
enlighten me. 
 A: The reason that $2,3$ are special in the “elliptic curve” case has to do with the dimension of elliptic curves: $1$. Elliptic curves are precisely the $1$-dimensional abelian varieties. If $X$ is an abelian variety of dimension $d$, the primes $p > 2d + 1$ will behave well, and likely the primes $p \le 2d + 1$ will be “bad”. For elliptic curves, $d = 1$, and this gives you precisely $2,3$ as bad primes.
You can read about this in [ST], and more precisely Corollary 2. They remark on the page following this corollary:

[…] Corrolary 2 suggests that, for abelian varieties of dimension $d$ (hence also for curves of genus $d$), it is the primes $p \le 2d+1$ which can play an especially nasty role. This is well-known for elliptic curves ($p = 2,3$), and the same set of bad primes seems to arise in other connections. For instance, a function field of one variable of genus $d$ is “conservative” if the characteristic $p$ is $> 2d + 1$.


[ST] — Jean-Pierre Serre and John Tate, Good Reduction of Abelian Varieties. The Annals of Mathematics, Second Series, Volume 88, Issue 3, (Nov., 1968), 492–517.
A: A good example why $p=2$ and $p=3$ are exceptional for Lie theory is the classification of simple modular Lie algebras over algebraically closed fields of characteristic $p$.
For $p=2$ a classification seems to be completely out of reach, and even for $p=3$ this might not be feasable. The reasons have been given already, but also trivial reasons from linear algebra play a role, e.g.,
$0=tr(A+A^t)=tr(A)+tr(A^t)=2tr(A)$ will not imply that $tr(A)=0$ for $p=2$.
However, for $p\ge 5$ the classification is very nice:
the simple Lie algebras fall into the following three classes:
$\bullet$ Finite dimensional graded Cartan type Lie algebras and their deformations.
$\bullet$ Classical Lie algebras $A_n,B_n,C_n,D_n,G_2,F_4,E_6,E_7,E_8$.
$\bullet$ The Melikyan algebras $M(m,n)$ on two parameters, for $p=5$.
A: The quote from Serre-Tate in jmc's answer indicates that for an abelian variety $A$ of dimension $d$, the primes that have the most complicated behavior are those satisfying $p\le 2d+1$, but it doesn't explain WHY those primes appear. The answer has to do with the representation of $\hbox{Gal}(\overline{\mathbb{Q}}/\mathbb{Q}$ on the Tate module, or more simply on the $\ell$-torsion $A[\ell]$. Things become more complicated, for example the computation of the conductor is quite complicated, if this representation is wildly ramified. And it is not that hard to prove (as is done in [ST]), by looking at the order of $\hbox{GL}_{2d}(\mathbb{Z}/\ell\mathbb{Z})$, that the representation can only be ramified at primes $p\le 2d+1$. For example, the gcd over all (sufficiently large) $\ell$ of $\#\hbox{GL}_{2}(\mathbb{Z}/\ell\mathbb{Z})$ is divisible only by 2 and 3, which is why they are the primes for elliptic curves that may be wildly ramified.
A: I think that in different theories, there is often a "primitive" fact (which is hard to explain further) that lies at the heart of the complication you mention. Let me give examples.
As for the "2 is the oddest prime" credo in number theory, often it boils down to the fact that $\mathbb{Q}$ contains exactly the second roots of unity. Or equivalently, the unit group of $\mathbb{Z}$ is $2$-torsion. I do not know if this can be embedded in a conceptual explanation; maybe it's a fact one has to live with, with ever-occuring consequences.
In the theory of algebraic groups and Lie algebras, e.g. in Chevalley bases and related stuff, the coefficients will be (or have as prime factors) only $2$ or $3$. A consequence is that many integral structures are $p$-integral only for primes $\ge 5$, and this pops up again and again in the theory. See Dietrich Burde's answer for more. I think here an ultimate explanation for this occurrence of $2$ and $3$ is that they appear in the basic combinatorics of root systems. That is the "primitive" fact.
As for the characteristic $2$ exception for quadratic forms, it is the non-equivalence of quadratic and symmetric bilinear forms that causes trouble. This in turn seems to be "primitive", just try to show equivalence and see that you have to invert $2$. And of course one should expect that for something quadratic, the number $2$ plays a special role.
I guess if we were more interested in some tri-linear stuff, or more in things that can be given as $7$-tuples than in pairs, the cases of characteristic $3$ or $7$ would need more attention. So this translates the question into why bilinear things, and pairs, are often natural. (Remark that such a basic thing as multiplication, including Lie brackets and other non-associative stuff, is a bilinear map and thus will have a tendency to need special treatment in characteristic $2$. Same for any duality, pairings etc.)
As for $2$ and $3$ as bad primes for elliptic curves, the story seems to be a little different. The answers by jmc and Joe Silverman suggest the following view:  there is a family of objects (abelian varieties) which can be parametrised roughly by certain numbers (dimension), and exceptional patterns are related to this parameter; and because elliptic curves are the ones where the parameter is small, there are small numbers that behave irregular. Now one would think that this is just a high-brow version of Alex Degtyarev's comment. But there is an interesting subtlety: It is not that in the general theory there are numbers different from $2, 3$ that misbehave (while these become nice), but there are more than just them. In other words: Granted that for every single number you might find some monstrosity somewhere in the general theory. But one might find it surprising that there are some numbers that always need care, even in the most specialised, well-behaved cases. For this, I have no better explanation than:
The strong law of small numbers: Small numbers (not necessarily primes) give exceptional patterns. Because naturally, there are so few of them, and they "have to satisfy too much at once". Maybe this is as far as one gets if one seeks after a common pattern between the "primitive" explanations above.
