I think that in different theories, there is often a "primitive" fact (which is hard to explain further) that lies at the heart of the complication you mention. Let me give examples.

As for the "2 is the oddest prime" credo in *number theory*, often it boils down to the fact that $\mathbb{Q}$ contains exactly the second roots of unity. Or equivalently, the unit group of $\mathbb{Z}$ is $2$-torsion. I do not know if this can be embedded in a conceptual explanation; maybe it's a fact one has to live with, with ever-occuring consequences.

In the theory of *algebraic groups and Lie algebras*, e.g. in Chevalley bases and related stuff, the coefficients will be (or have as prime factors) only $2$ or $3$. A consequence is that many integral structures are $p$-integral only for primes $\ge 5$, and this pops up again and again in the theory. See Dietrich Burde's answer for more. I think here an ultimate explanation for this occurrence of $2$ and $3$ is that they appear in the basic combinatorics of root systems. That is the "primitive" fact.

As for the characteristic $2$ exception for *quadratic forms*, it is the non-equivalence of quadratic and symmetric bilinear forms that causes trouble. This in turn seems to be "primitive", just try to show equivalence and see that you have to invert $2$. And of course one should expect that for something quadratic, the number $2$ plays a special role.

I guess if we were more interested in some tri-linear stuff, or more in things that can be given as $7$-tuples than in pairs, the cases of characteristic $3$ or $7$ would need more attention. So this translates the question into why bilinear things, and pairs, are often natural. (Remark that such a basic thing as *multiplication*, including Lie brackets and other non-associative stuff, is a bilinear map and thus will have a tendency to need special treatment in characteristic $2$. Same for any duality, pairings etc.)

As for $2$ and $3$ as bad primes for *elliptic curves*, the story seems to be a little different. The answers by jmc and Joe Silverman suggest the following view: there is a family of objects (abelian varieties) which can be parametrised roughly by certain numbers (dimension), and exceptional patterns are related to this parameter; and because elliptic curves are the ones where the parameter is small, there are small numbers that behave irregular. Now one would think that this is just a high-brow version of Alex Degtyarev's comment. But there is an interesting subtlety: It is not that in the general theory there are numbers **different** from $2, 3$ that misbehave (while these become nice), but there are **more** than just them. In other words: Granted that for every single number you might find some monstrosity somewhere in the general theory. But one might find it surprising that there are some numbers that *always* need care, even in the most specialised, well-behaved cases. For this, I have no better explanation than:

*The strong law of small numbers*: Small numbers (not necessarily primes) give exceptional patterns. Because naturally, there are so few of them, and they "have to satisfy too much at once". Maybe this is as far as one gets if one seeks after a common pattern between the "primitive" explanations above.