Stable sheaves are *simple*, i.e., $\textrm{End}E\simeq \mathbb{C}$. One thing that you want to avoid is the jumping of the automorphism group in a family.

A classical example is to consider a hyperelliptic curve $X$, and $[L]\in\textrm{Pic}^{g-1}X$.
If $\pi:X\to \mathbb{P}^1$ is the $g^1_2$, then Grothendieck-Riemann-Roch plus Riemann-Hurwitz tell you that
$\pi_\ast L\simeq \mathcal{O}(a-1)\oplus \mathcal{O}(-a-1)$,
where $a=h^0(L)$.
So you can take a take a family of line bundles over the unit disk $\{L_t\}_{t\in\Delta}$,
with $h^0(L_0)=1$, $h^0(L_t)=0$ for $t\in\mathbb{C}^\ast$. Then the generic element will be
semistable, $\mathcal{O}_{\mathbb{P}^1}(-1)^{\oplus 2}$, with automorphism group $GL_2$, and over zero you have $\mathcal{O}\oplus \mathcal{O}(-2)$, unstable, with 5-dimensional automorphism group.

And of course you need boundedness, see Donu's comment.

**ADDENDUM**

Here is an example of how allowing unstable bundles messes up uniqueness of limits
(and hence separatedness).
Let $X$ be a curve of genus $g\geq 2$, and let $E$ be a semi-stable rank two bundle
with $\det E\simeq \mathcal{O}_X$. Let $[L]\in \textrm{Pic}^d X$, $d\geq 2g$.
Then $E\otimes L$ is semi-stable of determinant $L^2$. It is globally generated
and surjects onto $L^2$, and so $E$ fits in an extension
$$
0\longrightarrow L^{-1}\longrightarrow E\longrightarrow L\longrightarrow 0.
$$
Now, take a DVR $R$, $\textrm{Spec }R=\{p,0\}$, where $p$ is the generic point
and $0$ the closed point, and consider a family of bundles $\mathcal{F}$ over
$\textrm{Spec }R$, for which $\mathcal{F}_0\simeq E$. One can show that if $\mathcal{F}'$ is the elementary transformation of $\mathcal{F}$ along $L$, then
$\mathcal{F}_p'\simeq \mathcal{F}_p $, but $\mathcal{F}'_0$ fits in an extension
$$
0\longrightarrow L\longrightarrow \mathcal{F}'_0\longrightarrow L^{-1}\longrightarrow 0.
$$
However, by the choice of $L$, $H^1(X,L^2)=0$, so $\mathcal{F}'_0\simeq L\oplus L^{-1}$,
an unstable bundle.

allrank 2 degree 0 bundles on $P^1$ is unbounded. Consider the subfamily $\lbrace O(n)\oplus O(-n)\rbrace$ . – Donu Arapura Oct 10 '12 at 8:33