To construct a (coarse or fine) moduli space that is separated, one usually throw away some class of the object in question. For moduli of sheaves people talk about (semi-)stability. A coherent sheaf $E$ on a scheme $X$ is (semi-)stable if it is pure and for all subsheaf $F$ we have $p(F) <(\leq)\text{ } p(E)$, where $p$ is the reduced Hilbert polynomial with respect to some fixed polarization.

My question is, what's the $bad$ property of unstable sheaves so that one has to throw it away to get a $good$ moduli space? Or is this definition merely there to make GIT work?

-
GIT yes, certainly. But even before getting to this, one needs to make sure that the family is bounded i.e. parameterized by a variety of finite type. Observe that the family of all rank 2 degree 0 bundles on $P^1$ is unbounded. Consider the subfamily $\lbrace O(n)\oplus O(-n)\rbrace$ . – Donu Arapura Oct 10 '12 at 8:33

Stable sheaves are simple, i.e., $\textrm{End}E\simeq \mathbb{C}$. One thing that you want to avoid is the jumping of the automorphism group in a family.

A classical example is to consider a hyperelliptic curve $X$, and $[L]\in\textrm{Pic}^{g-1}X$. If $\pi:X\to \mathbb{P}^1$ is the $g^1_2$, then Grothendieck-Riemann-Roch plus Riemann-Hurwitz tell you that $\pi_\ast L\simeq \mathcal{O}(a-1)\oplus \mathcal{O}(-a-1)$, where $a=h^0(L)$. So you can take a take a family of line bundles over the unit disk $\{L_t\}_{t\in\Delta}$, with $h^0(L_0)=1$, $h^0(L_t)=0$ for $t\in\mathbb{C}^\ast$. Then the generic element will be semistable, $\mathcal{O}_{\mathbb{P}^1}(-1)^{\oplus 2}$, with automorphism group $GL_2$, and over zero you have $\mathcal{O}\oplus \mathcal{O}(-2)$, unstable, with 5-dimensional automorphism group.

And of course you need boundedness, see Donu's comment.

Here is an example of how allowing unstable bundles messes up uniqueness of limits (and hence separatedness). Let $X$ be a curve of genus $g\geq 2$, and let $E$ be a semi-stable rank two bundle with $\det E\simeq \mathcal{O}_X$. Let $[L]\in \textrm{Pic}^d X$, $d\geq 2g$. Then $E\otimes L$ is semi-stable of determinant $L^2$. It is globally generated and surjects onto $L^2$, and so $E$ fits in an extension $$0\longrightarrow L^{-1}\longrightarrow E\longrightarrow L\longrightarrow 0.$$ Now, take a DVR $R$, $\textrm{Spec }R=\{p,0\}$, where $p$ is the generic point and $0$ the closed point, and consider a family of bundles $\mathcal{F}$ over $\textrm{Spec }R$, for which $\mathcal{F}_0\simeq E$. One can show that if $\mathcal{F}'$ is the elementary transformation of $\mathcal{F}$ along $L$, then $\mathcal{F}_p'\simeq \mathcal{F}_p$, but $\mathcal{F}'_0$ fits in an extension $$0\longrightarrow L\longrightarrow \mathcal{F}'_0\longrightarrow L^{-1}\longrightarrow 0.$$ However, by the choice of $L$, $H^1(X,L^2)=0$, so $\mathcal{F}'_0\simeq L\oplus L^{-1}$, an unstable bundle.