Distance between two sets Let $A, B$ be two convex and closed subsets of $\mathbb{R}^n$. We would like to the minimum distance between these two sets. i.e., we want to find a solution for the following problem.
$$ \min \{||x-y||~ ~\hbox{for}~ x\in A, y\in B\}$$. 
I would like to see if there is a method to find a solution for this problem in general in the theory of optimization. 
 A: In the closed convex case there are some fairly efficient algorithms, as long as you can efficiently project any point $x$ onto $A$ and $B$. This class of algorithms (named alternating projections, and tracing back to von Neumann) iterate projecting first on $A$, then on $B$, and repeating until the sequence stabilizes. If the intersection between $A$ and $B$ is nonempty, then this sequence converges to some point in the intersection. If they don't intersect, but the distance between $A$ and $B$ is positive, the odd and even iterates converge respectively to the minimizers $x$ and $y$.
See http://en.wikipedia.org/wiki/Projections_onto_convex_sets for more details.
A: I'd like to give a little more general answer.
The problem of finding the minimum distance between two (convex) sets is a mainstay of the optimization theory. In fact, even finding the distance of a point to an hyper plane is such a problem. 
So in your case, given that $A,B$ are compact closed sets, your formulation can be solved fairly efficiently as long as you can express both sets in closed form. Indeed, the problem is a convex optimization problem, which can be solvable efficiently in many cases. 
For instance, if $A,B$ can be expressed in terms of the intersection of convex constraints, i.e. some  functions $g^i_B(),g_A^i()$, 
$min_{x,y} ||x-y||_2$
$ s.t.$
$\quad g_A^i(x)\leq 0\quad i=1,\ldots,n_A$
$\quad g_B^i(y)\leq 0\quad i=1,\ldots,n_B$
then any algorithm for general convex optimization can do. If also all the functions can be expressed in conic form (see Ben-Tal, A., & Nemirovski, A. (2001). Lectures on modern convex optimization: analysis, algorithms, and engineering applications) then the problem can be solved in polynomial time.
I myself have experienced solving some of this problem to test the optimization software produced the company I work in. Our solver, MOSEK,  deals with conic optimization, which includes a fairly large number of possible convex set you can think about. 
Of course, if your set is particularly structured or performance is a major issue, you must look for specialized algorithms. But as an advise, if you can I would use a general purpose solver as a basic, and usually robust, benchmark to compare against.
A: You are trying to solve what is known as a best approximation problem. 


*

*von Neumann's alternating projections does not work here (as might have been perhaps suggested above)

*You can use Dykstra's projection algorithm, which will find the desired projection (some reformulation will be needed though before you can apply it)

*See the paper by Bauschke, Combettes, and Luke on reflection methods---the references in that paper also put this problem in wider context. The "averaged reflections" method (which is essentially the Douglas-Rachford splitting scheme) often works better / faster than Dykstra's projection method.

A: This may help, as it applies to "arbitrary compact convex sets" in $\mathbb{R}^n$,
although primarily applied to polytopes:

Llanas, B., M. Fernández de Sevilla, and V. Feliú. "An iterative algorithm for finding a nearest pair of points in two convex subsets of $\mathbb{R}^n$." Computers & Mathematics with Applications, 40.8 (2000): 971-983. (Elsevier link)
Llanas, Bernardo. "Efficient computation of the Hausdorff distance between polytopes by exterior random covering." Computational Optimization and Applications 30.2 (2005): 161-194. (MathSciNet review)

Here is the MathSciNet review:
 
A: There is a much earlier (and seemingly very efficient) algorithm by Gilbert, Johnson, Kerthi. (1988, IEEE Journal of Robotics and Automation).
