One often hears in popular explanations of the failure to find a "Grand Unified Theory" that "Gravity goes off to infinity, but cutting off the edges gives us wrong answers", and other similar mathematically vague statements. Clearly, this issue has some kind of mathematical explanation, but I'm not really qualified to read the corresponding physics work, so I'm wondering what actually fails mathematically.

$\begingroup$ Tolland's answer is a very nice exposition of nonrenormalizability. However, I think you would be well served to read Chapter 1 of the book "Quantum Gravity" by Carlo Rovelli. He presents many answers to your question along with exposition of important ideas. Additionally, he introduces loop quantum gravity through spinfoams, which is a popular approach(especially due to another surge of progress in the past 2 years). If you want to go really deep down the rabbit hole, the sequel to that book is called "Modern Canonical Quantum General Relativity" by Thomas Thiemann. This is a serious book! $\endgroup$ – B. Bischof Jan 28 '10 at 15:13

6$\begingroup$ I hate to tell you, but spin foams are by no means a "popular" approach. The loop quantum gravity people are pretty much considered crackpots by the theoretical physics community. Their theories tend to badly break most, if not all, of the laws of physics. And while Rovelli used to apparently be a good scientist, it seems in the past 2030 or so years he hasn't done too much that's not crazy. These people have actually been laughed at during their talks in conferences for being totally nuts. $\endgroup$ – jeremy Jan 29 '10 at 0:39

15$\begingroup$ A fulltext search of the arxiv server finds 2,944 articles containing the phrase "loop quantum gravity". The field is active, with about one new preprint/day. Upon checking the twenty most recent preprints, in no case was the reference pejorative or critical; thus evaluations like "crackpot" and "totally nuts" may not reflect a consensus opinion. Moreover, the geometric toolset broadly associated to spin foams proves useful in practical systems engineering calculations of polarization transport; this is fun. Conclusion: it's prudent to form one's own opinion of its merits. $\endgroup$ – John Sidles Feb 25 '11 at 18:07
Other people have said that the problem is that GR isn't renormalizable. I want to explain what that means in measuretheoretic terms. What I say won't be 100% rigorous, but it should get the general story across.
Quantum field theories are generally defined using a Feynman path integral measure. This measn that you compute correlation functions of observables by summing over all histories of your system, weighting each history by e^{S}, where S is an functional on the space of fields, called the action. In a field theory, these histories are functions on some spacetime manifold.
Just as you define the ordinary integral as a limit of Riemann sums, you define the Feynman path integral as a limit of "regularized" path integral measures. There are a lot of ways to do this; one of the most popular is the lattice regularization. We choose a finite set of points in spacetime, living on a lattice whose nearest neighbors are a distance a apart. Then we choose a "microscopic" action on our space of fields and discretize it, replacing derivatives with finite difference quotients, and approximate the path integral's weighted sum over histories with a sum over functions defined on the lattice.
Let's grant for a moment that, for any fixed lattice spacing a, this procedure defines a quantum theory, meaning that you can use the moments of the measure to compute correlation functions for your observables. Let's also grant that the expectation values of observables satisfy classical equations of motion (so that when we approach the classical limits, and the probability distributions become concentrated at their expected value, we get deterministic evolution of these values).
We usually want to take a continuum limit, making the lattice spacing smaller and smaller. If we do this, while keeping fixed the coupling constants in the microscopic action we used to define our measure, in most cases, we run into a problem: the coupling constants in the classical equations of motion depend on the lattice spacing a, and become infinite as a goes to zero. So that's probably not the limit we want.
And indeed, in most circumstances, we know the classical physics, and are trying to find a quantum theory that reproduces it. So we'll try something else: make the coupling constants in the microscopic action depend on the lattice spacing, and hope that we can tune them in a way that keeps the classical physics fixed.
Sometimes this works; sometimes it doesn't. It can happen that there isn't any way of tuning the coupling constants so as to reproduce a nice classical limit; when this happens, the theory is said to be nonrenormalizable. This appears to be what happens in General Relativity, if you use the discretized EinsteinHilbert action as your microscopic action.

$\begingroup$ Thank you very much! I feel my understanding is finally becoming one connected piece, although very scarce. Maybe it is very late for a question, but I presume there are ways to regularize the theory other than the lattice method. If there are, one has to prove that they all give the same result in the end, or is it automatic given that the theory is renormalizable? $\endgroup$ – timur Dec 6 '10 at 20:55

1$\begingroup$ One would have to prove that different regularizations give the same limiting path integral measure. This should follow from the same arguments that demonstrate existence. But people mostly worry about existence for lattice theories at the moment; we don't have any other regularization schemes which work nearly as well. $\endgroup$ – userN Dec 6 '10 at 22:19
General relativity is nonrenormalizable.
[On Mariano's advice I edited this answer to incorporate some earlier comments...hopefully this is more coherent to read/SH]
What this actually means is that there is not a semigroup parametrized by some scale (length or wavenumber) that allows the equations of gravity at one scale to be rewritten as identicallooking equations with different parameters at another scale. The existence of such a semigroup is what renormalizability means. The semigroup is called the renormalization group.
The best way to understand renormalization intuitively is to consider the realspace renormalization of the 1D Ising model (or the 2D Ising model on a triangular lattice), or even simpler examples: for instance, an charged particle in an electrolyte attracts a shell of counterions, which in turn attract another shell of countercounterions, etc. The net effect is to transform the normal scaleinvariant potential into a scaledependent potential. This particular example is called Debye screening.
Although many nonabelian gauge theories (such as pure $SU(N)$ YangMills or the Standard Model) are renormalizable (although this has not been proven at a strictly mathematical degree of rigor, but see, e.g., BRST) and general relativity is also a (classical) gauge theory, the space of field configurations or gauge equivalence classes $A/G$ in GR is not well understood, principally because the gauge group (diffeomorphisms) is infinitedimensional. This also complicates attempts to take the approach of lattice gauge theory for GR.
Regarding the lattice approach: for a "nice" nonabelian gauge theory (such as the Standard Model) the gauge equivalence classes are better understood, but as I mentioned above, still not perfectly. Indeed, showing that discretized SU(2) quantum gauge theory has a welldefined limit (in particular, one that depends only on the size of the discretization and not its detailed structure) is half of a Millennium Problem. This has only been done at the level of rigor of mathematical physics, not mathematics.
Returning to the original focus, the easiest way to see that gravity is nonrenormalizable is the appearance of higherorder (< 4) terms in the action (this is called "power counting").

1$\begingroup$ That's not really a mathematical explanation. I'm looking for something more sophisticated. $\endgroup$ – Harry Gindi Jan 27 '10 at 23:48

5$\begingroup$ renormalization itself can be made pretty much mathematically precise, see ncatlab.org/nlab/show/renormalization $\endgroup$ – Urs Schreiber Jan 27 '10 at 23:54

1$\begingroup$ @Steve: you should edit the answer to include your comments. It makes for much more linear reading! :) $\endgroup$ – Mariano SuárezÁlvarez Jan 28 '10 at 0:43

1$\begingroup$ @Mariano: It beats "the level of rigor of physics"! $\endgroup$ – Steve Huntsman Jan 28 '10 at 1:42

2$\begingroup$ That's like saying "The bulletstopping power of a cardboard box!" I'm only teasing, of course. ;) $\endgroup$ – Harry Gindi Jan 28 '10 at 1:54
To add to what's been said, the modern idea of what a field theory comes from an effective field theory picture. That is, we look at a theory over some scale, and 'average out' the degrees of freedom smaller than that scale.
For example, you can take electromagnetism in a vacuum, which is well understood, and ask what happens in matter. In principle, for an exact description you'd have to describe the ~$10^{23}$ atoms making up whatever material you have in detail to understand their electric and magnetic fields, which means providing a full quantum mechanical description because the magnetic fields of media are `inherently quantum mechanical.'
But we don't have to actually do this because we average out all of the small scale behavior (essentially) by taking the continuum limit and making a few assumptions about the polarizability, etc. The interesting thing is that, while the microscopic physics is entirely determined by microscopic maxwell + the matter distribution, the macroscopic physics doesn't seem to strongly depend on anything `obviously' microscopic.
However, we DO see some mysterious new constants that show up in the effective picture of E&M we get out of this averaging procedure, eg, the magnetization or differing $\epsilon$ and $\mu$s from the free space case. Some of these show up as entirely new phenomena which are not evident in vacuum E&M, and some of them show up as something that looks just like vacuum E&M but with new constants.
The point of this is, the new effective theory has introduced new undetermined constants as a result of not knowing the smallscale physics.
Something similar happens in GR. We don't know a priori all of the smallscale physics because the details of how gravity has to interact with all of the standard model fields isn't determined in advance. So we write down an effective theory below some energy scale, and get some number of undetermined constants which `phenomenologically' encapsulate all of the unknown details. Then we say, we'd like to go to a smaller scale. So we write down a new effective theory. But since we don't know all of the details below a new scale, we end up writing down more unknown constants. As the scale goes to zero, we end up with an infinite number of undetermined constants.
It may not seem like it, but this is very closely related to what Tolland said above about the renormalization group.
The sense in which gravity is `nonrenormalizable' is the sense in which, in order to make everything match in order to be finite (which is the rule you use in the above procedure to make sure each step contains no infinities) you'd need to know an infinite number of corrections to the terms in your lagrangian. But these parameters are not determined any way other than by experiment.
So, to be clear, in some sense, there's nothing that "mathematically goes wrong" with gravity. The point of this procedure is to make sure quantum gravity would not give you any infinities, and it does not. But the cost of doing this is to make your theory depend on a countably infinite number of unknowns. It's just that there is no procedure to determine these constants simply by requiring consistency.
From another point of view, it's kind of like saying "I know the function f at two points" and wanting to find f. But all the information you have is that
$f(x) = \sum_n a_n x^n$.
If all you have is those two points, and the only requirement that $f$ be less than infinity, you can never ever determine $f$ exactly!

$\begingroup$ I like how you mention effective theories! Your example at the end seems a bit feeble, though. I usually visualize length scales as Fourier Transforms, i.e. an effective field theory gives equations for the first few Fourier coefficients while a microscopic theory fixes the other coefficients as well. Maybe there's a precise toy example? $\endgroup$ – Greg Graviton Feb 25 '11 at 19:58
Let me add a few comments that might fill in some gaps not covered by other answers.
 When would it be safe to say that gravity has been successfully quantized?
A quantum theory consists of an algebra of "observables" (taken often to be a C*algebra), a "state"positive linear functional on the algebra, and a way of assigning a Hermitian element of the algebra to each conceivable singlenumberoutcome experiment. The theoretical prediction for an experiment is given by evaluating the state on the corresponding observabletaking its "expectation value". Higher moments of the same observable give information on the statistical distribution of experimental outcomes under identical conditions. Of course, since physicists don't have perfect information about the state of the entire universe, we are forced to work with a class of states and test hypotheses about which are closest to fitting all available observations.
If the above construction can be carried out such that all experiments sensitive to gravitational effects have representative observables and all their expectation values with respect to states that are approximately classical reproduce the results of classical general relativity, then the result is a successful quantization of gravity.
 Has the above construction succeeded or failed for gravity any more or less than for any other field theory?
This construction has been mathematically rigorously successful for free fields on Minkowski spacetime (and one some classes of curved Lorentzian manifolds, aka spacetimes). Other examples of rigorous constructions exist, but they are not of direct physical relevance. The golden standard of physically relevant field theories is quantum electrodynamics (QED). It has not yet been rigorously constructed. However, assuming that QED exists as a quantum theory and that all expectation values of observables in it can be expanded as power series in the strength of the interaction (electron charge), then each coefficient can be constructed order by order. Existence of a family of quantum theories corresponding to such a series expansion has not been resolved at the mathematical level.
The above construction is referred to as "perturbation theory". Essentially, it's a procedure that starts from a rigorously constructed free theory and constructs an interacting theory, in the above sense, as a power series in a finite number of parameters (coupling constants, which measure the strength of the interaction). This transition from free to interacting (for a given set of coupling constants) is not necessarily unique. Theories for which this nonuniqueness is parametrized by a finite number of parameters are called renormalizable, the rest are called nonrenormalizable.
QED and the rest of the standard model have been found to be renormalizable, but perturbative gravity, obtained by expanding around Minkowski spacetime, has been found to be nonrenormalizable. In principle, the nonuniqueness in the perturbative construction of interacting field theories needs to be fixed from experimental input. The unbounded number of experimental inputs needed to construct a particular realization of a nonrenormalizable field theory is what makes such theories unattractive.
An even bigger failure on the part of gravity is that the classical structure of general relativity (a Lorenzian manifold) enters in an essential way into the construction of free quantum field theories. If it is omitted, we have no way of guaranteeing the correct classical limit of the resulting quantum theory. On the other hand, a Lorentzian manifold should be the result of a classical limit of quantized gravity, not an input to it. This problem has not yet been resolved at the level of theoretical physics.
 What role do Euclidean functional/path integrals play?
The answer to this question is that it is unclear. Euclidean path integrals are equivalent to other constructions of quantum field theories if the spacetime manifold has translational symmetry along the time direction. In such cases, given a Euclidean path integral formulation of a field theory (as mentioned in A.J. Tolland's reply), the corresponding quantum theory can be constructed and vice versa.
However, in the case of the gravitational path integral, if it is used as a starting point for a nonperturbative definition, it may not correspond to a quantum theory at all. Since the functional integration is performed over all possible Lorentzian metrics, there is no background timetranslation symmetry to speak of. Therefore, even the construction of the Euclidean path integral succeeds, the equivalence to a quantum theory may fail. Again, this question has not yet been resolved even at the level of theoretical physics.
 Are discretization or other nontrivial modifications of the underlying manifold structure of spacetime necessary?
Unknown. However, there is no direct evidence that gravity cannot be formulated as just another field theory on a smooth manifold. In fact, there are counter examples: the infinitely many possibilities for perturbative quantization of gravity on Minkowski spacetime. Recall that nonrenormalizability does not mean nonexistence, just nonuniqueness. The many proposals that try to do away with a smooth manifold are heuristic at best.
If anyone is interested, I can elaborate on any of the points I've brought up above.

1$\begingroup$ Nice post. I would add though that nonrenormalizability obliterates any predictive power in field theories. $\endgroup$ – Steve Huntsman Jan 28 '10 at 19:21

5$\begingroup$ It does not remove predictive power necessarily. If you're only interested in a low enough energy regime where all but finitely many of the infinite constants are small enough to ignore, you have a perfectly good theory. Eg, the four fermi theory, or gravity to one loop level, are both perfectly good effective (nonrenormalizable) QFTs. $\endgroup$ – jeremy Jan 29 '10 at 0:35

4$\begingroup$ Most physicists believe that QED does NOT exist as a quantum theory because it is not asymptotically free and thus has bad behavior at short distances (e.g. a Landau pole). QCD, the theory of the strong interactions, is asymptotically free and is quite likely to exist by itself as a well defined quantum theory without needing any additional degrees of freedoms at short distances. QED may be the gold standard in terms of comparison with experiment, but QCD is the gold standard in terms of physically relevant quantum field theories which are most likely to be mathematically well defined. $\endgroup$ – Jeff Harvey Jan 19 '11 at 3:23

$\begingroup$ In the section "When would it be safe to say that gravity has been successfully quantized?" you seem to ignore the case of a spacetime that is not globally hyperbolic. How does your definition of success deal with this case? $\endgroup$ – Kelly Davis Feb 25 '11 at 20:22

$\begingroup$ @Kelly, you could say that I also ignored the case of spacetimes that are globally hyperbolic, since I mentioned neither one kind nor the other. The notion of global hyperbolicity relies on the metric as a classical field. When the metric is a quantum field, it is not clear how this notion generalizes. This remark is implicit in the last paragraph of my second bullet point. In case this notion does generalize, I see no reason for exclusion of "quantum non globally hyperbolic" spacetimes for prior mathematical reasons. OTOH, physical objections could always be raised, as is done classically. $\endgroup$ – Igor Khavkine Feb 26 '11 at 0:33
I'd say that the really fundamental issue is that the two theories have such different flavors that it's impossible to come up with an interpolation. General Relativity is about Lorentzian manifolds, so the objects are smooth, and the geometry is paramount to understanding the physics. On the other hand, although to do quantum mechanics does require vector bundles (or more generally, principal $G$bundles for $G=SU(3)\times SU(2)\times U(1)$ in particular) the study of these things is fundamentally rooted in discrete objects: the root, weight and character lattices of lie groups, which are where charges live, the collection of Feynman diagrams for a theory, are very much combinatorial.
One thing bearing this out is that two of the most well known attempts at quantum gravity work very hard to interpolate between these: in string theory, instead of looking at graphs, you have Riemann surfaces, which in some sense smooths everything out to be more like relativity, but in loop quantum gravity, you end up with discrete spacetime, and I've heard people talk about things like "Smooth discrete spaces" which I know absolutely nothing about, but which seem like a decent candidate for the cosmos in quantum gravity.

6$\begingroup$ Charles, this isn't right. You can treat gravity as a quantum field theory; the problem is that has to be an "effective" field theory, with a built in cutoff. If you try to remove the cutoff, you get nonsense, but there's nothing intrinsically wrong with a cutoff theory; Fermi's weak interaction theory has a cutoff. $\endgroup$ – userN Jan 28 '10 at 0:35

3$\begingroup$ @AJ, I'm aware that GR can be seen as a QFT, with gravity as a field on flat space, but doing so is like trying to look at a Monet as a bunch of dots on paper, it's missing the point entirely. Just because a theory can be described without geometry, doesn't mean it isn't a fundamentally geometric theory, and that QFT is fundamentally not. $\endgroup$ – Charles Siegel Jan 28 '10 at 2:31

3$\begingroup$ Charles, this line of logic proves too much; it applies with equal ease to classical YangMills theory. $\endgroup$ – userN Jan 28 '10 at 2:59

3$\begingroup$ I would like to point out that loop quantum gravity is taken seriously only by about 6 or 7 not well respected people in the physics community ;). Every known formulation of it breaks about all the laws of physics, unfortunately, which we've been trying to tell them for about 20 or 30 years... But the big things are that it breaks lorentz invariance, unitarity, and does not take place in an actual hilbert space. If you want references there's a good one on arxiv by peeters from a few years ago, and a bunch of others from the past few decades that aren't hard to find if you look. $\endgroup$ – jeremy Jan 28 '10 at 10:05

4$\begingroup$ This answer has many incorrect statements. For example, "...to do quantum mechanics does require vector bundles..." This is false. The standard particleinabox does not require a vector bundle. It continues "...to do quantum mechanics does require vector bundles (or more generally, principal Gbundles for G=SU(3)×SU(2)×U(1) in particular)..." This statement, besides being at best misleading, makes me think you do not know the difference between quantum mechanics, quantum field theory, and the standard model. These are three different things. $\endgroup$ – Kelly Davis Feb 25 '11 at 21:27
"I'm wondering what actually fails mathematically. "
It is the physical model that fails. The physical model leads to certain equations that do not have physical (as well as mathematical) solutions. Even zerothorder approximation fails.
This property does not belong exclusively to QG. Any QFT fails because of bad physical model. It is seen first in the zerothorder approximation (it is too "far" form the exact solution); next it is seen as divergences of perturbative corrections (IR and UV infinities).
Normally one cannot proceed without patching and repairing the originally obtained solutions. Sometimes such "doctoring" numbers works but it is just a huge luck and in the end this luck turns into a bad luck  people (even mathematicians!) get used to patching and repairing solutions (???!!!) and try to apply such "prescriptions" to all models. Fortunately it does not work in many cases so not all people get smug.
There are several ideologies of justifying "renormalization" but even they fail. Nevertheless many keep smiling and "explaining" even in that case ("effective theory" ideology) like a failed magician.
It is practically impossible to find a reasonable attitude to all that. I was trying to understand and explain "success" of renormalizations myself and I even succeeded but my work needs further development.

$\begingroup$ A question to jeremy: can the tree level QED be considered as "prefect effective theory"? I mean, the first Born approximation results such as Mott, Bhabha, KelinNishina cross sections, are they OK in you opinion? $\endgroup$ – Vladimir Kalitvianski Feb 26 '11 at 11:45