As a follow up to me other question, what can be said about unstable vector bundles? I know this is rather open ended, but what sorts of horrible things does having a subbundle of strictly greater slope imply?

1$\begingroup$ A nonhorrible thing: it has a "HarderNarasimhan filtration" by smaller bundles, where each quotient is stable. Instead of making a moduli space of stable vector bundles, you could make a moduli space of vector bundles with fixed dimensions in the HN filtration. See AtiyahBott's YangMills paper for more. $\endgroup$ – Allen Knutson Dec 3 '10 at 1:52
Well, I don't know about horrible. There's a lot you can say that's good! I'll start rambling and see where I end up.
I'm going to pretend you said principal GL(n)bundle instead of rank n vector bundle. Same thing, really, since we have the standard representation.
The collection Bun(n,C) of all principal GL(n) bundles P on a smooth curve C is a very nice geometric object: it's an Artin stack. It's not connected; the different components are labelled by topological data, like the Chern class. The tangent "space" (complex, really) to Bun(n,C) at a point P is naturally the derived global sections RGamma(C,ad(P)), where ad(P) is the associated bundle with fiber the adjoint representation of GL(n). The zeroth cohomology gives the infinitesimal automorphisms and the 1st cohomology gives the deformations. So the stabilizer group of any point V in Bun(n,C) is finitedimensional, and the dimension of the stack is n(g1) (by RiemannRoch). Bun(n,C) is smooth, and unobstructed, thanks to the vanishing of H^2(C,ad(P)).
Bun(n,C) has a very nice stratification, too. It's an increasing union of quotient stacks [A/G] of projective varieties by finitedimensional groups. Roughly, A is the stack of pairs (P,t), where t is a trivialization of P in an infinitesimal neighborhood of some point in C. Make the neighborhood large enough, i.e., rth order, and you can kill off all the automorphisms of P. Unfortunately, except for n=1, there is no uniform bound on r that works for all bundles. So, Bun(n,C) isn't a finite type quotient stack.
You can also realize Bun(n,C) (homotopically) as the infinite type quotient stack of U(n) connections modulo complexified gauge transformations. That's what Atiyah & Bott do in their paper "The YangMills Equations on Riemann Surfaces". (They also have a nice discussion of slopestability and the stratification.)
The top component of the stratification (those bundles where the stabilizer group is as small as possible) is the stack of (semi)stable vector bundles. If you take the coarse moduli space of this substack, you get the usual moduli space of stable bundles.
In summary: If you drop the stability conditions, you get a lot more geometry with a similar flavor, and without the random bits of weirdness that crop up in the theory of moduli spaces. (e.g., the stack always carries a universal bundle, you don't need the rank and the chern class to be coprime.)
OK, I'll stop evangelizing now.

$\begingroup$ Well, I'm actually quite fine with Artin stacks and such. Here, I was more thinking about what happens when you take a specific unstable bundle, and what you can say about it, and also partly why (other than to get a GIT quotient) it might be bad. However, thanks for doing a few things explicitly that I'd been meaning to work through. $\endgroup$ – Charles Siegel Oct 21 '09 at 3:05

$\begingroup$ Heh. I did say I'd ramble, not that I'd answer your question. But you might say that what's distinctive about unstable bundles is that their automorphism groups can get bigger. (Caution: misleading when n=1) $\endgroup$ – userN Oct 21 '09 at 11:58

$\begingroup$ " It's an increasing union of quotient stacks [A/G] of projective varieties by finitedimensional groups." Shouldn't it be "quasiprojective varieties"? (Or maybe something strange is going on here.) $\endgroup$ – t3suji Dec 3 '10 at 2:43

$\begingroup$ Yes, thanks! In particular the statement I have in mind is the one gotten by considering trivializations of the bundles over formal neighborhoods of points. $\endgroup$ – userN Dec 3 '10 at 4:17
Reider's proof of the Fujita conjecture that on a smooth surface, K+4A is very ample whenever A is ample, relies fundamentally on the Bogomolov instability theorem. I.e. It is precisely the Bogomolov instability of the vector bundle produced by Serre's construction associated to a possible base point of a linear series K+L, under certain conditions on the chern classes, that allows one to conclude the result. This technique seems to have played a major role in the study of linear series since that time. See Igor Reider's paper in the Annals of Math, (1988), or the article of Lazarsfeld in volume 3 of the IAS Park City series, Complex algebraic geometry.
If you prefer to phrase it as bad behavior related to instability, I guess you could, since this shows that the presence of something bad, namely a base point, implies instability, which forces something else (the presence of certain curves though the base point on the surface) which if not bad, is at least rather special. And this lets you classify all cases where the original bad behavior occurs. So certain specific bundles can only be unstable in rather unusual ways.
Another reason behind the notion of stability is the "jumping phenomenon." Namely, you can construct a family of vector bundles parametrized by the disk where all the fibers apart from the origin are mutually isomorphic, but not isomorphic to the fiber at the origin. Concretely, you can realize this by scaling an extension class. In the AtiyahBottKempfNess picture, this corresponds to nonclosed orbits of the complex gauge group. When the base manifold has dimension greater than one, the pieces in the HarderNarasimahan filtration may not be subbundles, so the limiting object is in general a torsionfree sheaf, and not necessarily a bundle.

$\begingroup$ this is a good example for why stability is related to good behavior in regard to moduli. I.e. this shows the moduli space may not be separated at non stable bundles. $\endgroup$ – roy smith Dec 3 '10 at 3:17