What's "bad" about unstable sheaves? To construct a (coarse or fine) moduli space that is separated, one usually throw away some class of the object in question. For moduli of sheaves people talk about (semi-)stability. A coherent sheaf $E$ on a scheme $X$ is (semi-)stable if it is pure and for all subsheaf $F$ we have $p(F) <(\leq)\text{ } p(E)$, where $p$ is the reduced Hilbert polynomial with respect to some fixed polarization.
My question is, what's the $bad$ property of unstable sheaves so that one has to throw it away to get a $good$ moduli space? Or is this definition merely there to make GIT work?
 A: The standard explanation is that if we want to work on the level of the coarse moduli space and not on the level of the stack itself, we need to tame the automorphism groups of objects (as Peter pointed out) and have some control over the maps between objects in your moduli. Semi-stable objects, Harder-Narasimhan and Jordan-Holder filtrations are exactly the notions supplying the rigidity for the morphisms between objects.
In this way it seems that the need to impose certain stability conditions spans  beyond moduli of sheaves, e.g. (just one other case I am familiar with) to have a good moduli space one considers (semi)-stable representations of quivers.
A. Rudakov axiomatises the situation in the case of an arbitrary abelian category, see his paper "Stability for an Abelian Category".
A: Stable sheaves are simple, i.e., $\textrm{End}E\simeq \mathbb{C}$. One thing that you want to avoid is the jumping of the automorphism group in a family.
A classical example is to consider a hyperelliptic curve $X$, and $[L]\in\textrm{Pic}^{g-1}X$.
If $\pi:X\to \mathbb{P}^1$ is the $g^1_2$, then Grothendieck-Riemann-Roch plus Riemann-Hurwitz tell you that 
$\pi_\ast L\simeq \mathcal{O}(a-1)\oplus \mathcal{O}(-a-1)$, 
where $a=h^0(L)$.
So you can take a take a family of line bundles over the unit disk $\{L_t\}_{t\in\Delta}$,
with $h^0(L_0)=1$, $h^0(L_t)=0$ for $t\in\mathbb{C}^\ast$. Then the generic element will be
semistable, $\mathcal{O}_{\mathbb{P}^1}(-1)^{\oplus 2}$, with automorphism group $GL_2$, and over zero you have $\mathcal{O}\oplus \mathcal{O}(-2)$, unstable, with 5-dimensional automorphism group.
And of course you need boundedness, see Donu's comment.
ADDENDUM
Here is an example of how allowing unstable bundles messes up uniqueness of limits
(and hence separatedness). 
Let $X$ be a curve of genus $g\geq 2$, and let $E$ be a semi-stable rank two bundle
with $\det E\simeq \mathcal{O}_X$. Let $[L]\in \textrm{Pic}^d X$, $d\geq 2g$.
Then $E\otimes L$ is semi-stable of determinant $L^2$. It is  globally generated
and surjects onto $L^2$, and so $E$ fits in an extension
$$
0\longrightarrow L^{-1}\longrightarrow E\longrightarrow L\longrightarrow 0.
$$
Now, take a DVR $R$, $\textrm{Spec }R=\{p,0\}$, where $p$ is the generic point
and $0$ the closed point, and consider a family of bundles $\mathcal{F}$ over
$\textrm{Spec }R$, for which $\mathcal{F}_0\simeq E$.  One can show that if $\mathcal{F}'$ is the elementary transformation of $\mathcal{F}$ along $L$, then 
$\mathcal{F}_p'\simeq \mathcal{F}_p $, but $\mathcal{F}'_0$ fits in an extension
$$
0\longrightarrow L\longrightarrow \mathcal{F}'_0\longrightarrow L^{-1}\longrightarrow 0.
$$
However, by the choice of $L$, $H^1(X,L^2)=0$, so $\mathcal{F}'_0\simeq L\oplus L^{-1}$,
an unstable bundle.
A: To make GIT work is not a good reason. There are constructions of moduli spaces that do not use GIT (mainly, because in certain situations GIT does not work). Boundedness as Donu notes is a better reason. In other words it's not that unstable (in itself) is "bad", but that (semi-)stable is "good". Yet in other words, one "bad" property of unstable sheaves is that there are too many of them.
