Is there a high-concept explanation for why characteristic 2 is special? The structure of the multiplicative groups of $\mathbb{Z}/p\mathbb{Z}$ or of $\mathbb{Z}_p$ is the same for odd primes, but not for $2.$ Quadratic reciprocity has a uniform statement for odd primes, but an extra statement for $2$. So in these examples characteristic $2$ is a messy special case.
On the other hand, certain types of combinatorial questions can be reduced to linear algebra over $\mathbb{F}_2,$ and this relationship doesn't seem to generalize to other finite fields. So in this example characteristic $2$ is a nice special case.
Is anything deep going on here? (I have a vague idea here about additive inverses and Fourier analysis over $\mathbb{Z}/2\mathbb{Z}$, but I'll wait to see what other people say.)
 A: When I was a lad, I was taught that 2 acted strange (compared to other prime numbers) because it was of the form $1-u$ for a unit, $u$. I guess the test of whether this holds water would be something like: let $R$ be a (commutative) ring (with unity), then do the irreducibles in $R$ of the form $1-u$, $u$ a unit in $R$, stand out from the other irreducibles in $R$ in some significant way? 
I think there is some evidence in favor of this hypothesis in the rings ${\bf Z}[\rho]$, where $\rho$ is a $p$th root of unity, and $1-\rho$ is an irreducible that may require special consideraiton. 
A: I think there are two phenomena at work, and often one can separate behaviors based on whether they are "caused by''one or the other (or both). One phenomenon is the smallness of $2$, i.e., the expression $p-1$ shows up when describing many characteristic $p$ and $p$-adic structures, and the qualitative properties of these structures will change a lot depending on whether $p-1$ is one or greater than one. For example:


*

*Adding a primitive $p^\text{th}$ root of unity $z$ to ${\bf Q}_p$ yields a totally ramified field extension of degree $p-1$. The valuation of $1-z$ is $1/(p-1)$ times the valuation of $p$. This is a long way of saying that $-1$ lies in ${\bf Q}_2$.

*The group of units in the prime field of a characteristic $p$ field has order $p-1$.  This is the difference between triviality and nontriviality.

*As you mentioned, some combinatorial questions can be phrased in Boolean language and attacked with linear algebra.


The other phenomenon is the evenness of $2$. Standard examples:


*

*Negation has a nontrivial fixed point.  This gives one way to explain why there are $4$ square roots of $1 \pmod {2^n}$ (for $n$ large), but only $2$ in the $2$-adic limit.  If you combine this with smallness, you find that negation does nothing, and this adds a lot of subtlety to the study of algebraic groups (or generally, vector spaces with forms).

*The Hasse invariant is a weight $p-1$ modular form, and odd weight forms behave differently from even weight forms, especially in terms of lifting to characteristic zero, level 1. This is a bit related to David's mention of abelian varieties — I've heard that some Albanese "varieties" in characteristic $2$ are non-reduced.

A: I see binary arithmetic to be the natural companion to set theory, and this singles $\mathbb{F}_2$ out given that set theory is the core of mathematics. The basic idea being that all subsets of a finite set with n elements can be associated with a binary $n$-tuple (an element of $(\mathbb{Z}_2)^n$). Or vice-versa, since we could as well consider set theory as the study of binary $n$-tuples. (Just an elementary example: the sum of two such $n$-tuples, using $1+1=0$, corresponds to the symmetric difference of the two sets). The very fact that a set of $n$ elements has $2^n$ subsets reminds us of the core meaning of the powers of $2$.
A: Maybe this isn't very high concept, but I've always thought the "original sin" of $2$ was that there's a integer which is a second root of unity, which doesn't happen for any other prime.
Why is this deep?  Well, one way to think of it as this: in fields of characteristic $p$, $p^\text{th}$ roots of unity must all be trivial (and in general, taking $p^\text{th}$ roots is a bad idea), so fields of characteristic $2$ are particularly incompatible with the integers, since they have to destroy $-1$.
A: This is certainly not very high-concept, but I've always just thought there was more "room to maneuver" in high characteristic, and there's some inexplicable (well, nearly inexplicable!) "phase change" that happens between characteristic 2 and 3. And yeah, this is fuzzy and not well-defined, but it's nevertheless how I think about it?
A: I think one reason that makes 2 special is that, for the only archimedean place of the rationals, namely the real numbers, it has absolute Galois group of order 2.
A: I think $2$ is not special, we just see the weirdness at $2$ earlier than the weirdness at odd primes.
For example, consider $\operatorname{Ext}_{E(x)}(\mathbb{F}_p , \mathbb{F}_p)$ where $E(x)$ denotes an exterior algebra over $\mathbb{F}_p.$ If $p=2$ this is a polynomial algebra on a class $x_1$ in degree $1$ and if $p$ is odd this is an exterior algebra on a class $x_1$ tensor a polynomial algebra on $x_2$. I say these are the same, generated by $x_1$ and $x_2$ in both cases and with a $p$-fold Massey product $\langle x_1,\dotsc,x_1 \rangle = x_2.$ The only difference is that a $2$-fold Massey product is simply a product.
In what sense are the $p$-adic integers $\mathbb{Z}_p$ the same? One way to say it is that if you study the algebraic $K$-theory of $\mathbb{Z}_p$ you find that the first torsion is in degree $2p-3$. If $p=2$ this is degree $1$, and $K_1(A)$ measures the units of $A$ (for a reasonable ring $A$). If $p$ is odd it measures something something more complicated. Another way to say it is that $\mathbb{Z}_p$ is the first Morava stabilizer algebra and there is something special about the $n^\text{th}$ Morava stabilizer algebra at $p$ if $p-1$ divides $n$. If you study something like topological modular forms, this means the primes $2$ and $3$ are special.
The dual Steenrod algebra is generated by $\xi_i$ at $p=2$ and by $\xi_i$ and $\tau_i$ at odd primes. But really it is generated by $\tau_i$ with a $p$-fold Massey product $\langle\tau_i,\dotsc,\tau_i\rangle = \xi_{i+1}$ at all primes, after renaming the generators at $p=2$. (Again a $2$-fold Massey product is just a product.)
I could go on, but maybe this is enough for now.
A: I think the simplest answer is just that:  $\mathbb{F}_2$ is much simpler to analyze and use, and for example has the concrete interpretation of also being a nice way to have permutation matrices defined over a field, as well as being a cute way of encoding statements in boolean logic.
In short, I don't think any of this is emblematic something deep, but rather that $\mathbb{F}_2$ is the easiest to understand of all the finite fields, and its simplicity makes it very useful in works that call for a finite field 
A: $x\mapsto x^2$ is a (1-1) automorphism on fields of characteristic 2, whereas it is 2-1 on $F_q\setminus{0}$ if q is odd. Not a high level concept, but this is where all things quadratic (reciprocity, residues, etc.) break down.
A: It's anthropocentrism. If we were starfish, we would think that the prime $5$ was the weird one.
A: Most of the examples of characteristic $2$ being funny that I know of (and there are a lot) really boil down to the fact that $x = -x$.
A: Here is my computational reason (instead of a high-concept explanation) why the primes 2 and 3 are special (hypotheses of many theorems on algebraic groups, linear or projective exclude both these primes). 
The numbers 2 and 3  got into the Primes Club by 'dubious' means!
Given $p$, to certify it as a prime a number, we  need to check no number $d $ with $1 < d \leq [ \sqrt p ]$ divides it. For 2 and 3 this condition is vacuously true as there are no integers in that interval, wheres 5 onwards they really needed to pass the test!
