Stable sheaves are simple, i.e., $\textrm{End}E\simeq \mathbb{C}$. One thing that you want to avoid is the jumping of the automorphism group in a family.

A classical example is to consider a hyperelliptic curve $X$, and $L\in\textrm{Pic}^{g-1}X$. If $\pi:X\to \mathbb{P}^1$ is the $g^1_2$, then Grothendieck-Riemann-Roch plus Riemann-Hurwitz tell you that $\pi_\ast L\simeq \mathcal{O}(a-1)\oplus \mathcal{O}(-a-1)$, where $a=h^0(L)$. So you can take a take a family of line bundles over the unit disk $\{L_t\}_{t\in\Delta}$, with $h^0(L_0)=1$, $h^0(L_t)=0$ for $t\in\mathbb{C}^\ast$. Then the generic element will be semistable, $\mathcal{O}_{\mathbb{P}^1}(-1)^{\oplus 2}$, with automorphism group $GL_2$, and over zero you have $\mathcal{O}\oplus \mathcal{O}(-2)$, unstable, with 5-dimensional automorphism group.

And of course you need boundedness, see Donu's comment.

ADDENDUM

Here is an example of how allowing unstable bundles messes up uniqueness of limits (and hence separatedness). Let $X$ be a curve of genus $g\geq 2$, and let $E$ be a semi-stable rank two bundle with $\det E\simeq \mathcal{O}_X$. Let $[L]\in \textrm{Pic}^d X$, $d\geq 2g$. Then $E\otimes L$ is semi-stable of determinant $L^2$. It is globally generated and surjects onto $L^2$, and so $E$ fits in an extension $$ 0\longrightarrow L^{-1}\longrightarrow E\longrightarrow L\longrightarrow 0. $$ Now, take a DVR $R$, $\textrm{Spec }R=\{p,0\}$, where $p$ is the generic point and $0$ the closed point, and consider a family of bundles $\mathcal{F}$ over $\textrm{Spec }R$, for which $\mathcal{F}_0\simeq E$. One can show that if $\mathcal{F}'$ is the elementary transformation of $\mathcal{F}$ along $L$, then $\mathcal{F}_p'\simeq \mathcal{F}_p $, but $\mathcal{F}'_0$ fits in an extension $$ 0\longrightarrow L\longrightarrow \mathcal{F}'_0\longrightarrow L^{-1}\longrightarrow 0. $$ However, by the choice of $L$, $H^1(X,L^2)=0$, so $\mathcal{F}'_0\simeq L\oplus L^{-1}$, and unstable bundle.

Stable sheaves are simple, i.e., $\textrm{End}E\simeq \mathbb{C}$. One thing that you want to avoid is the jumping of the automorphism group in a family.

A classical example is to consider a hyperelliptic curve $X$, and $[L]\in\textrm{Pic}^{g-1}X$. If $\pi:X\to \mathbb{P}^1$ is the $g^1_2$, then Grothendieck-Riemann-Roch plus Riemann-Hurwitz tell you that $\pi_\ast L\simeq \mathcal{O}(a-1)\oplus \mathcal{O}(-a-1)$, where $a=h^0(L)$. So you can take a take a family of line bundles over the unit disk $\{L_t\}_{t\in\Delta}$, with $h^0(L_0)=1$, $h^0(L_t)=0$ for $t\in\mathbb{C}^\ast$. Then the generic element will be semistable, $\mathcal{O}_{\mathbb{P}^1}(-1)^{\oplus 2}$, with automorphism group $GL_2$, and over zero you have $\mathcal{O}\oplus \mathcal{O}(-2)$, unstable, with 5-dimensional automorphism group.

And of course you need boundedness, see Donu's comment.

ADDENDUM

Here is an example of how allowing unstable bundles messes up uniqueness of limits (and hence separatedness). Let $X$ be a curve of genus $g\geq 2$, and let $E$ be a semi-stable rank two bundle with $\det E\simeq \mathcal{O}_X$. Let $[L]\in \textrm{Pic}^d X$, $d\geq 2g$. Then $E\otimes L$ is semi-stable of determinant $L^2$. It is globally generated and surjects onto $L^2$, and so $E$ fits in an extension $$ 0\longrightarrow L^{-1}\longrightarrow E\longrightarrow L\longrightarrow 0. $$ Now, take a DVR $R$, $\textrm{Spec }R=\{p,0\}$, where $p$ is the generic point and $0$ the closed point, and consider a family of bundles $\mathcal{F}$ over $\textrm{Spec }R$, for which $\mathcal{F}_0\simeq E$. One can show that if $\mathcal{F}'$ is the elementary transformation of $\mathcal{F}$ along $L$, then $\mathcal{F}_p'\simeq \mathcal{F}_p $, but $\mathcal{F}'_0$ fits in an extension $$ 0\longrightarrow L\longrightarrow \mathcal{F}'_0\longrightarrow L^{-1}\longrightarrow 0. $$ However, by the choice of $L$, $H^1(X,L^2)=0$, so $\mathcal{F}'_0\simeq L\oplus L^{-1}$, an unstable bundle.

Stable sheaves are simple, i.e., $\textrm{End}E\simeq \mathbb{C}$. One thing that you want to avoid is the jumping of the automorphism group in a family.

A classical example is to consider a hyperelliptic curve $X$, and $L\in\textrm{Pic}^{g-1}X$. If $\pi:X\to \mathbb{P}^1$ is the $g^1_2$, then Grothendieck-Riemann-Roch plus Riemann-Hurwitz tell you that $\pi_\ast L\simeq \mathcal{O}(a-1)\oplus \mathcal{O}(-a-1)$, where $a=h^0(L)$. So you can take a take a family of line bundles over the unit disk $\{L_t\}_{t\in\Delta}$, with $h^0(L_0)=1$, $h^0(L_t)=0$ for $t\in\mathbb{C}^\ast$. Then the generic element will be semistable, $\mathcal{O}_{\mathbb{P}^1}(-1)^{\oplus 2}$, with automorphism group $GL_2$, and over zero you have $\mathcal{O}\oplus \mathcal{O}(-2)$, unstable, with 5-dimensional automorphism group.

And of course you need boundedness, see Donu's comment.

Stable sheaves are simple, i.e., $\textrm{End}E\simeq \mathbb{C}$. One thing that you want to avoid is the jumping of the automorphism group in a family.

A classical example is to consider a hyperelliptic curve $X$, and $L\in\textrm{Pic}^{g-1}X$. If $\pi:X\to \mathbb{P}^1$ is the $g^1_2$, then Grothendieck-Riemann-Roch plus Riemann-Hurwitz tell you that $\pi_\ast L\simeq \mathcal{O}(a-1)\oplus \mathcal{O}(-a-1)$, where $a=h^0(L)$. So you can take a take a family of line bundles over the unit disk $\{L_t\}_{t\in\Delta}$, with $h^0(L_0)=1$, $h^0(L_t)=0$ for $t\in\mathbb{C}^\ast$. Then the generic element will be semistable, $\mathcal{O}_{\mathbb{P}^1}(-1)^{\oplus 2}$, with automorphism group $GL_2$, and over zero you have $\mathcal{O}\oplus \mathcal{O}(-2)$, unstable, with 5-dimensional automorphism group.

And of course you need boundedness, see Donu's comment.

ADDENDUM

Here is an example of how allowing unstable bundles messes up uniqueness of limits (and hence separatedness). Let $X$ be a curve of genus $g\geq 2$, and let $E$ be a semi-stable rank two bundle with $\det E\simeq \mathcal{O}_X$. Let $[L]\in \textrm{Pic}^d X$, $d\geq 2g$. Then $E\otimes L$ is semi-stable of determinant $L^2$. It is globally generated and surjects onto $L^2$, and so $E$ fits in an extension $$ 0\longrightarrow L^{-1}\longrightarrow E\longrightarrow L\longrightarrow 0. $$ Now, take a DVR $R$, $\textrm{Spec }R=\{p,0\}$, where $p$ is the generic point and $0$ the closed point, and consider a family of bundles $\mathcal{F}$ over $\textrm{Spec }R$, for which $\mathcal{F}_0\simeq E$. One can show that if $\mathcal{F}'$ is the elementary transformation of $\mathcal{F}$ along $L$, then $\mathcal{F}_p'\simeq \mathcal{F}_p $, but $\mathcal{F}'_0$ fits in an extension $$ 0\longrightarrow L\longrightarrow \mathcal{F}'_0\longrightarrow L^{-1}\longrightarrow 0. $$ However, by the choice of $L$, $H^1(X,L^2)=0$, so $\mathcal{F}'_0\simeq L\oplus L^{-1}$, and unstable bundle.