For a smooth $Y$, a necessary condition for contractibility is that the conormal line bundle $N_{Y,X}^*$ is ample. It is also sufficient for contracting to an algebraic space. The reference is Algebraization of formal moduli. II. Existence of modifications. by M. Artin.
$Y$ can be contracted to a point on an algebraic (projective) variety if in addition $Y=\mathbb P^{n-1}$, $n=\dim X$. You can prove this easily by hands. Start with an ample divisor $H$ and then prove that an appropriate linear combination $|aH+bY|$ is base point free and is zero exactly on $Y$. You will find the argument in Matsuki's book on Mori's program for example.
So if $X$ is a surface and $Y=\mathbb P^1$ with $Y^2<0$ then it is contractible to a projective surface. For a reducible divisor $Y=\sum Y_i$ a necessary condition (which is also sufficient in the category of algebraic spaces) is that the matrix $(Y_i.Y_j)$ is negative definite. The strongest elementary sufficient condition for contractibility to a variety is that $\sum Y_i$ is a rational configuration of curves. This is contained in On isolated rational singularities of surfaces by M. Artin.
This paper also contains an example of an elliptic curve $Y$ with $Y^2=-1$ which is not contractible to an algebraic surface. The surface $X$ is the blowup of $\mathbb P^2$ at 10 sufficiently general points lying on a smooth cubic, $Y$ is the strict preimage of that cubic.
Finally, for an irreducible divisor $Y$ the resulting space $V$ is smooth iff $Y=\mathbb P^{n-1}$ and $N_{Y,X}=\mathcal O(-1)$. Indeed, $X\to V$ has to factor through the blowup of $V$ at a point by the universal property of the blowup. But then $X$ has to coincide with this blowup by Zariski main theorem. And on the blowup at a point the exceptional divisor is $\mathbb P^{n-1}$ with the normal bundle $\mathcal O(-1)$.