One often hears in popular explanations of the failure to find a "Grand Unified Theory" that "Gravity goes off to infinity, but cutting off the edges gives us wrong answers", and other similar mathematically vague statements. Clearly, this issue has some kind of mathematical explanation, but I'm not really qualified to read the corresponding physics work, so I'm wondering what actually fails mathematically.

Other people have said that the problem is that GR isn't renormalizable. I want to explain what that means in measuretheoretic terms. What I say won't be 100% rigorous, but it should get the general story across. Quantum field theories are generally defined using a Feynman path integral measure. This measn that you compute correlation functions of observables by summing over all histories of your system, weighting each history by e^{S}, where S is an functional on the space of fields, called the action. In a field theory, these histories are functions on some spacetime manifold. Just as you define the ordinary integral as a limit of Riemann sums, you define the Feynman path integral as a limit of "regularized" path integral measures. There are a lot of ways to do this; one of the most popular is the lattice regularization. We choose a finite set of points in spacetime, living on a lattice whose nearest neighbors are a distance a apart. Then we choose a "microscopic" action on our space of fields and discretize it, replacing derivatives with finite difference quotients, and approximate the path integral's weighted sum over histories with a sum over functions defined on the lattice. Let's grant for a moment that, for any fixed lattice spacing a, this procedure defines a quantum theory, meaning that you can use the moments of the measure to compute correlation functions for your observables. Let's also grant that the expectation values of observables satisfy classical equations of motion (so that when we approach the classical limits, and the probability distributions become concentrated at their expected value, we get deterministic evolution of these values). We usually want to take a continuum limit, making the lattice spacing smaller and smaller. If we do this, while keeping fixed the coupling constants in the microscopic action we used to define our measure, in most cases, we run into a problem: the coupling constants in the classical equations of motion depend on the lattice spacing a, and become infinite as a goes to zero. So that's probably not the limit we want. And indeed, in most circumstances, we know the classical physics, and are trying to find a quantum theory that reproduces it. So we'll try something else: make the coupling constants in the microscopic action depend on the lattice spacing, and hope that we can tune them in a way that keeps the classical physics fixed. Sometimes this works; sometimes it doesn't. It can happen that there isn't any way of tuning the coupling constants so as to reproduce a nice classical limit; when this happens, the theory is said to be nonrenormalizable. This appears to be what happens in General Relativity, if you use the discretized EinsteinHilbert action as your microscopic action. 


General relativity is nonrenormalizable. [On Mariano's advice I edited this answer to incorporate some earlier comments...hopefully this is more coherent to read/SH] What this actually means is that there is not a semigroup parametrized by some scale (length or wavenumber) that allows the equations of gravity at one scale to be rewritten as identicallooking equations with different parameters at another scale. The existence of such a semigroup is what renormalizability means. The semigroup is called the renormalization group. The best way to understand renormalization intuitively is to consider the realspace renormalization of the 1D Ising model (or the 2D Ising model on a triangular lattice), or even simpler examples: for instance, an charged particle in an electrolyte attracts a shell of counterions, which in turn attract another shell of countercounterions, etc. The net effect is to transform the normal scaleinvariant potential into a scaledependent potential. This particular example is called Debye screening. Although many nonabelian gauge theories (such as pure $SU(N)$ YangMills or the Standard Model) are renormalizable (although this has not been proven at a strictly mathematical degree of rigor, but see, e.g., BRST) and general relativity is also a (classical) gauge theory, the space of field configurations or gauge equivalence classes $A/G$ in GR is not well understood, principally because the gauge group (diffeomorphisms) is infinitedimensional. This also complicates attempts to take the approach of lattice gauge theory for GR. Regarding the lattice approach: for a "nice" nonabelian gauge theory (such as the Standard Model) the gauge equivalence classes are better understood, but as I mentioned above, still not perfectly. Indeed, showing that discretized SU(2) quantum gauge theory has a welldefined limit (in particular, one that depends only on the size of the discretization and not its detailed structure) is half of a Millennium Problem. This has only been done at the level of rigor of mathematical physics, not mathematics. Returning to the original focus, the easiest way to see that gravity is nonrenormalizable is the appearance of higherorder (< 4) terms in the action (this is called "power counting"). 


To add to what's been said, the modern idea of what a field theory comes from an effective field theory picture. That is, we look at a theory over some scale, and 'average out' the degrees of freedom smaller than that scale. For example, you can take electromagnetism in a vacuum, which is well understood, and ask what happens in matter. In principle, for an exact description you'd have to describe the ~$10^{23}$ atoms making up whatever material you have in detail to understand their electric and magnetic fields, which means providing a full quantum mechanical description because the magnetic fields of media are `inherently quantum mechanical.' But we don't have to actually do this because we average out all of the small scale behavior (essentially) by taking the continuum limit and making a few assumptions about the polarizability, etc. The interesting thing is that, while the microscopic physics is entirely determined by microscopic maxwell + the matter distribution, the macroscopic physics doesn't seem to strongly depend on anything `obviously' microscopic. However, we DO see some mysterious new constants that show up in the effective picture of E&M we get out of this averaging procedure, eg, the magnetization or differing $\epsilon$ and $\mu$s from the free space case. Some of these show up as entirely new phenomena which are not evident in vacuum E&M, and some of them show up as something that looks just like vacuum E&M but with new constants. The point of this is, the new effective theory has introduced new undetermined constants as a result of not knowing the smallscale physics. Something similar happens in GR. We don't know a priori all of the smallscale physics because the details of how gravity has to interact with all of the standard model fields isn't determined in advance. So we write down an effective theory below some energy scale, and get some number of undetermined constants which `phenomenologically' encapsulate all of the unknown details. Then we say, we'd like to go to a smaller scale. So we write down a new effective theory. But since we don't know all of the details below a new scale, we end up writing down more unknown constants. As the scale goes to zero, we end up with an infinite number of undetermined constants. It may not seem like it, but this is very closely related to what Tolland said above about the renormalization group. The sense in which gravity is `nonrenormalizable' is the sense in which, in order to make everything match in order to be finite (which is the rule you use in the above procedure to make sure each step contains no infinities) you'd need to know an infinite number of corrections to the terms in your lagrangian. But these parameters are not determined any way other than by experiment. So, to be clear, in some sense, there's nothing that "mathematically goes wrong" with gravity. The point of this procedure is to make sure quantum gravity would not give you any infinities, and it does not. But the cost of doing this is to make your theory depend on a countably infinite number of unknowns. It's just that there is no procedure to determine these constants simply by requiring consistency. From another point of view, it's kind of like saying "I know the function f at two points" and wanting to find f. But all the information you have is that $f(x) = \sum_n a_n x^n$. If all you have is those two points, and the only requirement that $f$ be less than infinity, you can never ever determine $f$ exactly! 


Let me add a few comments that might fill in some gaps not covered by other answers.
A quantum theory consists of an algebra of "observables" (taken often to be a C*algebra), a "state"positive linear functional on the algebra, and a way of assigning a Hermitian element of the algebra to each conceivable singlenumberoutcome experiment. The theoretical prediction for an experiment is given by evaluating the state on the corresponding observabletaking its "expectation value". Higher moments of the same observable give information on the statistical distribution of experimental outcomes under identical conditions. Of course, since physicists don't have perfect information about the state of the entire universe, we are forced to work with a class of states and test hypotheses about which are closest to fitting all available observations. If the above construction can be carried out such that all experiments sensitive to gravitational effects have representative observables and all their expectation values with respect to states that are approximately classical reproduce the results of classical general relativity, then the result is a successful quantization of gravity.
This construction has been mathematically rigorously successful for free fields on Minkowski spacetime (and one some classes of curved Lorentzian manifolds, aka spacetimes). Other examples of rigorous constructions exist, but they are not of direct physical relevance. The golden standard of physically relevant field theories is quantum electrodynamics (QED). It has not yet been rigorously constructed. However, assuming that QED exists as a quantum theory and that all expectation values of observables in it can be expanded as power series in the strength of the interaction (electron charge), then each coefficient can be constructed order by order. Existence of a family of quantum theories corresponding to such a series expansion has not been resolved at the mathematical level. The above construction is referred to as "perturbation theory". Essentially, it's a procedure that starts from a rigorously constructed free theory and constructs an interacting theory, in the above sense, as a power series in a finite number of parameters (coupling constants, which measure the strength of the interaction). This transition from free to interacting (for a given set of coupling constants) is not necessarily unique. Theories for which this nonuniqueness is parametrized by a finite number of parameters are called renormalizable, the rest are called nonrenormalizable. QED and the rest of the standard model have been found to be renormalizable, but perturbative gravity, obtained by expanding around Minkowski spacetime, has been found to be nonrenormalizable. In principle, the nonuniqueness in the perturbative construction of interacting field theories needs to be fixed from experimental input. The unbounded number of experimental inputs needed to construct a particular realization of a nonrenormalizable field theory is what makes such theories unattractive. An even bigger failure on the part of gravity is that the classical structure of general relativity (a Lorenzian manifold) enters in an essential way into the construction of free quantum field theories. If it is omitted, we have no way of guaranteeing the correct classical limit of the resulting quantum theory. On the other hand, a Lorentzian manifold should be the result of a classical limit of quantized gravity, not an input to it. This problem has not yet been resolved at the level of theoretical physics.
The answer to this question is that it is unclear. Euclidean path integrals are equivalent to other constructions of quantum field theories if the spacetime manifold has translational symmetry along the time direction. In such cases, given a Euclidean path integral formulation of a field theory (as mentioned in A.J. Tolland's reply), the corresponding quantum theory can be constructed and vice versa. However, in the case of the gravitational path integral, if it is used as a starting point for a nonperturbative definition, it may not correspond to a quantum theory at all. Since the functional integration is performed over all possible Lorentzian metrics, there is no background timetranslation symmetry to speak of. Therefore, even the construction of the Euclidean path integral succeeds, the equivalence to a quantum theory may fail. Again, this question has not yet been resolved even at the level of theoretical physics.
Unknown. However, there is no direct evidence that gravity cannot be formulated as just another field theory on a smooth manifold. In fact, there are counter examples: the infinitely many possibilities for perturbative quantization of gravity on Minkowski spacetime. Recall that nonrenormalizability does not mean nonexistence, just nonuniqueness. The many proposals that try to do away with a smooth manifold are heuristic at best. If anyone is interested, I can elaborate on any of the points I've brought up above. 


I'd say that the really fundamental issue is that the two theories have such different flavors that it's impossible to come up with an interpolation. General Relativity is about Lorentzian manifolds, so the objects are smooth, and the geometry is paramount to understanding the physics. On the other hand, although to do quantum mechanics does require vector bundles (or more generally, principal $G$bundles for $G=SU(3)\times SU(2)\times U(1)$ in particular) the study of these things is fundamentally rooted in discrete objects: the root, weight and character lattices of lie groups, which are where charges live, the collection of Feynman diagrams for a theory, are very much combinatorial. One thing bearing this out is that two of the most well known attempts at quantum gravity work very hard to interpolate between these: in string theory, instead of looking at graphs, you have Riemann surfaces, which in some sense smooths everything out to be more like relativity, but in loop quantum gravity, you end up with discrete spacetime, and I've heard people talk about things like "Smooth discrete spaces" which I know absolutely nothing about, but which seem like a decent candidate for the cosmos in quantum gravity. 


It is the physical model that fails. The physical model leads to certain equations that do not have physical (as well as mathematical) solutions. Even zerothorder approximation fails. This property does not belong exclusively to QG. Any QFT fails because of bad physical model. It is seen first in the zerothorder approximation (it is too "far" form the exact solution); next it is seen as divergences of perturbative corrections (IR and UV infinities). Normally one cannot proceed without patching and repairing the originally obtained solutions. Sometimes such "doctoring" numbers works but it is just a huge luck and in the end this luck turns into a bad luck  people (even mathematicians!) get used to patching and repairing solutions (???!!!) and try to apply such "prescriptions" to all models. Fortunately it does not work in many cases so not all people get smug. There are several ideologies of justifying "renormalization" but even they fail. Nevertheless many keep smiling and "explaining" even in that case ("effective theory" ideology) like a failed magician. It is practically impossible to find a reasonable attitude to all that. I was trying to understand and explain "success" of renormalizations myself and I even succeeded but my work needs further development. 

