Given a finitedimensional semisimple complex Lie algebra $\mathfrak{g}$, the BernsteinGelfandGelfand category $\mathcal O$ is the full subcategory of $\mathfrak g$modules satisfying some finiteness conditions. It contains all finitedimensional modules as well as all highestweight modules, it's Noetherian and Artinian, and it's Abelian. It's clear to me why you would want to work in some full subcategory of $\mathfrak g$modules which has the above properties, but why $\mathcal O$? Is it minimal in some sense with respect to these properties and/or some other important properties?

8$\begingroup$ The answers below are very good and give nice, thorough explanations of the use of category O. I would add one small comment, which is that category O also contains a lot of important modules: the Verma modules (which are universal highestweight modules), the ring of functions on the unipotent radical U of your chosen Borel inside of a group G with lie algebra g, the enveloping algebra of Lie U^, etc. If you want to learn more about how those modules behave, considering them as objects inside of cat O and doing homological algebra there can be quite useful. $\endgroup$– Chuck HagueMay 15, 2011 at 18:10
3 Answers
I'd add to what Ben says the observation that people have found a number of different module categories valuable for different purposes within this same Lie algebra context. (And some Lie theory people don't really find category $\mathcal{O}$ to be all that important in their own work.) Even two people close to the original construction, Joseph Bernstein and Sergei Gelfand, found it more useful to broaden the study to categories satisfying somewhat different finiteness conditions in their further work on projective functors.
Much of the original motivation for category $\mathcal{O}$ came from a rethinking of the classical finite dimensional theory combined with an attempt to understand better the problems raised by Verma's thesis and later work by Jantzen. Here is where the translation functors really come into their own, along with the refined use of central characters and blocks. But the "correct" module category to study depends on which problems are being studied. The category of all modules for a universal enveloping algebra is definitely too big for practical purposes, but within it there are many attractive subcategories.
P.S. As these answers illustrate, there can be several different answers to the "why' question asked. The answers have certainly evolved over time, as illustrated in part by the series of BGG and BG papers. For example, the BGG category turns out to be just right for BGG Reciprocity, due to the special nature of projective objects in this category relative to Verma modules. (On the other hand, the earlier prototype of BGG Reciprocity in prime characteristic involved just finite dimensional modules and therefore required just the natural category of such modules for the finite dimensional restricted enveloping algebra of the Lie algebra of a semisimple algebraic group.)

6$\begingroup$ I wonder which pair of people have the largest number of posts where they have given the only two answers. We must be extremely high on the list. $\endgroup$– Ben Webster ♦May 13, 2011 at 22:08
It might be worth pointing out a different motivation for Category O, namely the theory of Harish Chandra (g,K) modules. These are algebraic models for continuous representations of real reductive groups (the real forms of g). Harish Chandra's amazing theory reduces many questions in the representation theory of real groups to this algebraic theory, which can then be studied geometrically, for example by BeilinsonBernstein localization.
In any case Category O is essentially the category of Harish Chandra modules associated to the complex reductive group G, when considered as a real Lie group. There are slight subtleties in what kind of semisimplicity/local finiteness etc we require for the center of the enveloping algebra or the maximal torus, but in broad strokes the two coincide, and category O is thus a nice combinatorially accessible model of a very basic object in representation theory of Lie $groups$.

3$\begingroup$ I think this is a good answer, but the difference between category O and HC bimodules does sweep quite a bit of structure under the rug.... $\endgroup$– Ben Webster ♦May 17, 2011 at 22:57

1$\begingroup$ @BenWebster Sorry, could you expand on the difference/similarity of HC bimodules and category O in my question about this for quantizations of symplectic singularities mathoverflow.net/questions/352074/… ? Is this related to the question math.stackexchange.com/questions/2369680/… and could you maybe answer that question as well? $\endgroup$ Feb 6, 2020 at 17:41
It's actually not the minimal category with those properties. For example, the subcategory of category O where the center of $U(\mathfrak{g})$ acts semisimply is smaller and satisfies all of those. It does become essentially minimal (I think you also want to impose closed under passing to sub or quotient objects) if you also require that the subcategory be closed under tensoring with finite dimensional representations. Such functors and their summands (translation functors) are ubiquitous in the study of category O, and I think can be credited with many of its good properties.

3$\begingroup$ I don't think the centre of $U(\mathfrak{g})$ acts trivially on all finite dimensional modules. $\endgroup$ May 14, 2011 at 4:30

2$\begingroup$ It does if you compensate for the generalized central character, actually; anyways, changed to "semisimply" to clarify. $\endgroup$– Ben Webster ♦May 14, 2011 at 5:14