Stable sheaves are simple, i.e., $\textrm{End}E\simeq \mathbb{C}$. One thing that you want to avoid is the jumping of the automorphism group in a family.
A classical example is to consider a hyperelliptic curve $X$, and $[L]\in\textrm{Pic}^{g-1}X$.
If $\pi:X\to \mathbb{P}^1$ is the $g^1_2$, then Grothendieck-Riemann-Roch plus Riemann-Hurwitz tell you that
$\pi_\ast L\simeq \mathcal{O}(a-1)\oplus \mathcal{O}(-a-1)$,
where $a=h^0(L)$.
So you can take a take a family of line bundles over the unit disk $\{L_t\}_{t\in\Delta}$,
with $h^0(L_0)=1$, $h^0(L_t)=0$ for $t\in\mathbb{C}^\ast$. Then the generic element will be
semistable, $\mathcal{O}_{\mathbb{P}^1}(-1)^{\oplus 2}$, with automorphism group $GL_2$, and over zero you have $\mathcal{O}\oplus \mathcal{O}(-2)$, unstable, with 5-dimensional automorphism group.
And of course you need boundedness, see Donu's comment.
ADDENDUM
Here is an example of how allowing unstable bundles messes up uniqueness of limits
(and hence separatedness).
Let $X$ be a curve of genus $g\geq 2$, and let $E$ be a semi-stable rank two bundle
with $\det E\simeq \mathcal{O}_X$. Let $[L]\in \textrm{Pic}^d X$, $d\geq 2g$.
Then $E\otimes L$ is semi-stable of determinant $L^2$. It is globally generated
and surjects onto $L^2$, and so $E$ fits in an extension
$$
0\longrightarrow L^{-1}\longrightarrow E\longrightarrow L\longrightarrow 0.
$$
Now, take a DVR $R$, $\textrm{Spec }R=\{p,0\}$, where $p$ is the generic point
and $0$ the closed point, and consider a family of bundles $\mathcal{F}$ over
$\textrm{Spec }R$, for which $\mathcal{F}_0\simeq E$. One can show that if $\mathcal{F}'$ is the elementary transformation of $\mathcal{F}$ along $L$, then
$\mathcal{F}_p'\simeq \mathcal{F}_p $, but $\mathcal{F}'_0$ fits in an extension
$$
0\longrightarrow L\longrightarrow \mathcal{F}'_0\longrightarrow L^{-1}\longrightarrow 0.
$$
However, by the choice of $L$, $H^1(X,L^2)=0$, so $\mathcal{F}'_0\simeq L\oplus L^{-1}$,
an unstable bundle.
