Why, from a string theory perspective, is it natural to consider the Deligne-Mumford (resp. Kontsevich) compactification of the moduli of curves (resp. maps [from curves to a target space X]) rather than some other compactification?
In any case, what other compactifications of the moduli of curves have been studied? Similarly, what other compactifications of the moduli of maps have been studied? Do any of these other compactifications lead to an interesting "Gromov-Witten theory"?
I can give individual answers to a lot of your questions, but I can't answer any of them completely, nor can I fit all these answers together into a coherent whole.
For string theory, there does seem to be something special about the Deligne-Mumford compactification. Morally, what's going on is this: string theorists are allowing cylinder-shaped submanifolds of their Riemann surfaces to become infinitely long. The only finite energy fields on such infinitely long submanifolds are constant, so you can replace the long cylinder with a node. (Likewise, morally, if you allow vertex operators at two marked points to come together, you should take their operator product. This is what bubbling when marked points collide does for you.)
Somewhat more technically: The first step in (bosonic) string theory is to compute the partition function of the nonlinear sigma model as a function on the space of metrics on your worldsheet. This function on metrics descends to a section of some line bundle on the moduli stack of complex structures on the worldsheet. When you can compute it at all, you can show that this section has exactly the right pole structure it needs to be a section of the 13th power of the canonical bundle tensored with the 2nd power of the dual of the line bundle corresponding to the boundary divisor of the Deligne-Mumford compactification. (There's an old Physics Report by Phil Nelson that explains this pretty well, although not with anything you'd call a proof. Should also credit Belavin & Knizhik, who did the initial calculations.)
There's a somewhat more modern perspective on this (Zwiebach, Sullivan, Costello,...) that says that the generating function of string theory correlation functions for smooth Riemann surfaces satisfies a certain equation (a "quantum master equation"), which gives instructions for how to extend the theory to nodal Riemann surfaces. Different master equations give different recipes for extending to the boundary, if I understand your advisor correctly.
People have played around with other compactifications. There are a lot of different compactifications of the stack of smooth marked curves. People have already mentioned a few of them. David Smyth has some cool results which classify the "stable modular" compactifications of the stack of curves (http://arxiv.org/abs/0902.3690 ). For compactifications of the moduli of maps, the only one that comes immediatley to mind is Losev, Nekrasov, and Shatashvili's "freckled instanton compactification", in which IIRC, you allow zeros and poles to collide and cancel each other out.
Here is a random thought on the second part of Kevin's question. There are various compactifications of the space of maps that should be meaningful physically but haven't been explored by the physicists. One example is Drinfeld's compactification. This is in some sense the smallest modular compactification of the space of maps - the points on the boundary have geometric meaning (Drinfeld calls them quasi-maps). It has the flavor of a gauged linear sigma model so it should be relevant to the physics somehow.
Here is how one can define Drinfeld's compactification. Suppose first that $G$ is a complex reductive group, and we want to study maps from $\mathbb{P}^{1}$ to the flag variety $G/P$ for some parabolic $P \subset G$. To compactify the space of such maps Drinfeld's beautiful idea is to look at a "compactification" of $G/P$, i.e. an Artin stack which contains $G/P$ as a dense open substack. There is a natural stack like that. If $R^{u}P$ is the unipotent radical of $P$, then the quasi-affine variety $V := G/R^{u}P$ is a principal $R$-bundle on $G/P$, where $R = P/R^{u}P$ is the maximal reductive quotient of $P$. We can now consider the affinization $W$ of $V$, i.e. $W = Spec(\Gamma(V,\mathcal{O}))$. Note that $V \subset W$ is zariski open, and the action of $R$ on $V$ automatically extends to $W$. The stack $[W/R]$ then contains $V/R = G/P$ as an open dense substack. The moduli of maps from $\mathbb{P}^{1}$ to $[W/R]$ such that the generic point of $\mathbb{P}^{1}$ maps to $G/P$ turns out to be compact. This moduli space is Drinfeld's compactification.
For instance if $G = SL_{2}$, then $V = \mathbb{C}^{2}-\{0\}$, $W = \mathbb{C}^{2}$, and $[W/R] = [\mathbb{C}^{2}/\mathbb{C}^{\times}]$.
To get Drinfeld's compactification for a general projective target $X$ we can embed $X$ in a projective space $\mathbb{P}^{N}$, and then close (i.e. take the fiber product) the space of maps from $\mathbb{P}^{1}$ to $X$ in Drinfeld's compactification of the space of maps from $\mathbb{P}^{1}$ to $\mathbb{P}^{N}$.
For flag variety targets Drinfeld's compactification of the space of maps is singular but it has a natural resolution - Laumon's space of quasi-flags. It is known that this resolution is semismall (this is a result of Kuznetsov). It is also known that Kontsevich's space of stable maps has a morphism onto Laumon's compactification. It seems to me, that it will be very interesting to study whether Laumon's compactification of maps from $\mathbb{P}^{1}$ to a projective variety has a virtual fundamental class. Since it is gotten by a fiber product with a map from a smooth space, Laumon's compactification will have a natural derived structure. I wonder if the obstruction theory given by this derived structure happens to be perfect.
This question is very much in the spirit of the paper of AJ with Frenkel and Teleman. Only here the target quotient stack is very special and maybe the question is easier to answer.
In low-dimensional topology, Thurston introduced a very interesting compactification (alas, I don't think this is at all connected to algebraic geometry, but it's beautiful and worth knowing!).
If M(g) is the moduli space of genus g curves, then you can express M(g) as the quotient of T(g) by the mapping class group, where T(g) is Teichmuller space. It is classical that T(g) is homeomorphic to an open ball in R^{6g-6}. Thurston compactified T(g) by the space of "measured foliations" (or, equivalently, "measured laminations") of the surface. The Thurston compactification of T(g) is homeomorphic to a closed ball in R^{6g-g} and is compatible with the mapping class group action, so it descends to an interesting topological compactification of moduli space.
Thurston used his compactification to prove the "Nielsen-Thurston" classification of surface homeomorphisms, which can be viewed as something like a "Jordan normal form" for surface homeomorphisms. Some information about this can be found in the following wikipedia article:
http://en.wikipedia.org/wiki/Nielsen-Thurston_classification
Another readable source of information about this is the manuscript "A Primer on Mapping Class Groups" by Farb and Margalit, which is available here :
Another compactification that's been studied a bit is the Satake compactification. Unlike the Thurston compactification, this is definitely algebraic geometry!
Satake introduced a natural compactification of the moduli space A(g) of principally polarized abelian varieties in genus g. Later, Walter Baily proved that this compactification turns A(g) into a projective variety. The construction uses theta functions and can be read about in Igusa's book "Theta Functions".
Anyway, let M(g) be the moduli space of curves. The map that takes an algebraic curve to its Jacobian induces a map M(g)-->A(g), and Torelli's theorem says that it is injective. Baily later showed that the closure of the image of M(g) in the Satake compactification of A(g) is also a projective variety compactifying M(g). Incidentally, this provided the first proof that M(g) is a quasiprojective variety!
I would prefer to leave this as a comment to Tony Pantev's answer, but I don't have enough reputation to do so. Anyway, as Tony mentions the Laumon compactification provides a semismall resolution of singularities of the Drinfeld compactification. However, to the best of my knowledge, the Laumon compactification is special to the case where G = SL_n, whereas the Drinfeld compactification can be defined for any reductive algebraic group. In Kuznetsov's paper, he says in the introduction that he would like to study the resolutions of the Drinfeld compactification for groups besides SL_n in a future paper. As far as I know, no such paper was ever written. Does anyone know if Kuznetsov ever wrote such a paper and, if not, if anyone else has ever worked on this?
One partial solution to this question seems like it might be buried in the paper by Braverman, Finkelberg, Gaitsgory, and Mirkovic where they compute the IC sheaves of the Drinfeld compactification. This is related because, assuming a semismall resolution of singularities existed, its cohomology would compute the intersection cohomology of the Drinfeld compactification. However, given their use of finite-dimensional Zastava spaces to model the singularities of the Drinfeld compactification (and hence to compute the IC sheaf), it does not seem to me that a semismall resolution can be found in their paper (although maybe this means that if a semismall resolution existed it should also provide a resolution of each of the Zastava spaces?).
If appropriate, I would be happy to start this as its own topic.
My understanding is that, in large part, it's because the compactification by stable curves/maps can be fairly easily constructed with GIT, and have been studied before. Most of the results people actually really want to have involve showing that the class of curves in something is actually contained in the locus of smooth curves (which really is the case for rational marked curves). The intro article by Fulton and Pandharipande discusses this a bit, in the case of homogeneous varieties.
Edit: In response to the answers mentioning Satake and Thurston: they don't have nice (that I know of) realizations as curves in the target space, which is somewhat important, to be able to really get your hands on what these extra points represent for the curves in a Calabi-Yau problem that enumerative geometry and string theory care about.
