Let $X,Y$ be analytic spaces with $X$ compact reduced and $Y$ arbitrary (i.e. maybe with nilpotents in its structure sheaf). Then Douady showed in his thesisthesis that the set of holomorphic maps $Hol(X,Y)$ can be endowed with the structure of an analytic space whose underlying topology is the compact-open topology. If $X,Y$ are compact manifolds, the Zariski tangent space at $f:X\to Y$ is a subspace of the finite-dimensional vector space of sections of the tangent bundle to $Y$ pulled-back to $X$ viz. $T_f(Hol(X,Y))\subset \Gamma(X,f^\star TY)$. I don't know any good general criterion for $Hol(X,Y)$ to be smooth at $f$.
Edit Here is a class of examples which might interest you, where smoothness occurs. Let $X$ be a Riemann surface of genus $g$. The space of ramified covers $f:X\to \mathbb P^1$ of degree $d$ is non-empty and smooth of dimension $2d+1-g$ as soon as $d\geq g+1$. But there are explicit cases for smaller $d$ where the corresponding space is singular. You can read about these results in this articlethis article by Akaohori and Namba.