The pentagon and hexagon axioms in the definition of a symmetric monoidal category are one example that I was thinking of here; the axioms of a weak 2-category are another. I understand that it can be checked laboriously that these few coherence axioms are sufficient to show, e.g. in the first case, that all coherence conditions we want on associativity and commutativity to hold do, but this is rather tedious. Is there some other motivation for the choice of coherence axioms?
I agree with Viritrilbia: there is in general no "nice" or canonical choice of coherence axioms. By "in general" I mean for an arbitrary 2-dimensional theory, such as the theory of weak 2-categories, monoidal categories, braided monoidal categories with duals, etc. This is true for the same reason that there is in general no "nice" or canonical presentation of a given group.
Nevertheless, there are some famous patterns in the coherence axioms for certain commonly encountered theories. The associahedra, or Stasheff polytopes, give the higher associativity coherence axioms. (There is one such polytope of each dimension. The pentagon is the associahedron of dimension 2.) The hexagon in the definition of symmetric monoidal category is also part of a general pattern.
Topologists often sweep units under the carpet. In the definition of monoidal category, the pentagon is not the only coherence axiom: there is also the triangle, which has to do with the unit coherence isomorphisms $X \otimes I \to X$. People talk much less about the triangle, even though it's a crucial part of the definition. I don't know of a general pattern that it fits into.
Incidentally, when the definition of monoidal category was first given (by Mac Lane?), there were one or two extra coherence axioms that were later shown to be redundant. This is an indication of how non-obvious the choice of axioms is.
Yes, this is closely related to the theory of operads. Here is a very informal discussion of the case of monoidal categories. In particular I will intentionally blur the distinction between spaces and groupoids.
A (non-symmetric) operad O is a gadget consisting of a bunch of spaces On which we think of parameterizing n-ary operations, together with structure that tells us how to compose operations. An algebra X over an operad O consists of a bunch of maps On x Xn -> X which are compatible with this composition structure. For example, the associative operad A has An = * for every n, so there's just one n-ary operation X^n -> X for each n, and this makes X into a monoid.
We would like to say that a monoidal category is a monoid object in categories, but this is too strict for most purposes. The problem in model category language is that the associative operad is not "cofibrant". What we need to do is find a "cofibrant replacement"--an operad B such that all the spaces Bn are still contractible, but in which those composition structure maps which I glossed over are better behaved. An example is the operad B formed from the associahedra. B2 is still a point, but B3 is an interval, and B4 is a pentagon. Now a B-algebra in categories consists of a category C together with a functor B2 x C x C = C x C -> C, a functor B3 x C x C x C -> C, a functor B4 x C4 -> C, etc. These functors are the monoidal product, the associator, and the pentagon identity respectively. There's nothing higher because the next bit of structure would be an "identity between identities", and we don't have any such thing in a category. But if we were defining a monoidal 2-category the pentagon identities be replaced by 2-morphisms called "pentagonators" and there would be a coherence condition coming from B5.
Edit: I should emphasize that we did not obtain the operad B from A in any canonical way--B was "pulled out of a hat". But the model category machinery ensures that if we had chosen a different cofibrant replacement B', then the notions of B-algebra and B'-algebra would be equivalent. This notion is thus associated to A in a canonical way; the familiar description is not canonical.
Sure, they come naturally from looking at higher-order rearrangements of products. For example, the pentagon axiom is exactly saying that there is only one way to go between any two ways of parenthesizing a product of four objects, and the hexagon axiom says the same for ways of permuting 3 objects. The morally correct definition is that the same holds if you replace four and three by any n, but this is not stated in the usual definition because it can be proven from just four and three.
Not really. There are techniques and heuristics for figuring out what the coherence axioms should be. But in a meaningful sense the "real" definition of, say, a monoidal category stipulates that all the coherence conditions hold, and it just happens to be convenient that a finite list of these suffice to prove the others.
There are, however, some general contexts in which general definitions and coherence theorems exist, for instance the notion of pseudoalgebras for a 2-monad. The notion of "unbiased monoidal category" is equivalent to pseudoalgebras for the 2-monad whose strict algebras are strict monoidal categories, and so its coherence axioms are determined and easy to write down. But to prove that ordinary "biased" monoidal categories are equivalent to unbiased ones is essentially the same as proving Mac Lane's original coherence theorem.