I know that abelian varieties of CM type have central importance in algebraic geomtry and number theory. There are many conjectures and concepts related to them like Andre-Oort, Coleman conjecture, Shimura varieties, etc... . But why CM abelian varieties came into consideration in first place? Why did mathematicians begin to study CM abelian varieties? I am in particular intereseted to know the relation to "special functions".
Why did mathematicians begin to study CM abelian varieties?
Because they could. The study of CM abelian varieties arguably starts with Fagnano's work on the length of the lemniscate. In 1799, Gauss linked the length of the lemniscate to the arithmetic-geometric mean and in the entries of Gauss's diary from 1814, including famously (but not restricted to) the last one, he studied some properties of biquadratic reciprocity and their links with the division of the lemniscate which he understood to derive from properties of Punktgitter (lattices of points) in the complex planes, i.e CM elliptic curves.
As you see, there is a continuous path from traditional questions in geometry arguably going back to Antiquity (Fagnano's motivation was to rectify the ellipse) to accounts of complex multiplication quite close to the modern version, and the latter was known (at least to Gauss) in the early XIXth century. Legendre, Jacobi, Abel, Galois all knew about CM from one perspective or another.
(An elaboration of Timo Keller's answer.)
One of the main sources of historical interest in CM abelian varieties is the pursuit of Kronecker's Jugendtraum and Hilbert's 12th problem. If $K$ is an imaginary quadratic field, then one can always find an elliptic curve $E$ with CM by the full ring of integers $\mathfrak o_K$ of $K$, and then it turns out that, (i) $K(j(E))$ is the Hilbert class field of $K$, and (ii) all the abelian extensions of $K$ can be constructed by adjoining to $K(j(E))$ values of a certain "modular function" (the Weber function $h$) evaluated at torsion points of $E$. For the precise statement and further elaboration, pick up any book that deals with the CM theory of elliptic curves, e.g., Silverman's second book on elliptic curves.
What about more general number fields $K$: can we explicitly construct their abelian extensions in a similar fashion? The answer to this general question is wide open. But, anyhow, one possible approach is to try to think about how to generalize the above picture, in which case we ask: what should replace the field $K$, the elliptic curve $E$, and the special functions $j$ and $h$?
Hecke gave the first attempt at answering these questions. In his thesis he takes up the case of a real quadratic $K$ and uses a certain "Hilbert modular function", but doesn't get far. Later on he tries his hand at dealing with certain biquadratic extensions, and constructs unramified abelian extensions of them by adjoining singular values of two-variable Hilbert modular functions. Implicitly, the geometric object taking the place of $E$ here is a certain abelian surface with CM. Next comes Shimura who takes this one step further and looks at a general CM number field $K$. His $E$ and special functions are then CM abelian varieties and certain "Siegel modular functions". He uses this setup to construct certain class fields, except not over $K$, but over a certain other field $K^*$, the so-called "reflex field". (In the case of elliptic curves, we have $K=K^*$, and we get all the class fields of $K$.) So, unfortunately, unlike in the case of elliptic curves and imaginary quadratic fields, the higher-dimensional theory is more intricate doesn't give as complete an answer. For more details, you can start with Shimura's Automorphic functions and number theory (LMN 1968).