What mathematical treatment is there on the renormalization group flow in a space of Lagrangians?

$\begingroup$ It's not clear to me what you're interested in. Have you read the appropriate Wikipedia page? en.wikipedia.org/wiki/Renormalization_group $\endgroup$– Ryan BudneyApr 23 '11 at 20:16

15$\begingroup$ This question could be vastly improved. Please add more background and details about what you're interested in and what you already know and what you'd like to know. Please also put a version of the question in the title. In fact, "What mathematical treatment is there on the renormalization group flow in a space of Lagrangians?" would make for a perfectly good title, and then you could have a few paragraphs of question. See mathoverflow.net/howtoask . $\endgroup$– Theo JohnsonFreydApr 23 '11 at 21:07
Dec 2017 edit: I just put a hopefully useful detailed complement to this post on https://physics.stackexchange.com/questions/372306/wilsoniandefinitionofrenormalizability
The answer to your question depends on whether you are interested in the perturbative RG or the nonperturbative one.
Typically one starts with a Gaussian measure $d\mu_{0,\infty}$ on a space of fields $\phi$ given by a covariance $$C_{0,\infty}(x,y)=\int \phi(x)\phi(y)\ d\mu_{0,\infty}(\phi)\ . $$ For instance one can take for the covariance $\frac{1}{\xi^2}$ in Fourier space. Then one introduces a UV regularization at length scale $l$ by multiplying for instance by $\exp(l^2 \xi^2)$ which cutsoff momenta $\xi$ which are larger than $l^{1}$. This defines $$ C_{l,\infty}(x,y)=\frac{1}{(2\pi)^d}\int_{\mathbb{R}^d} \frac{e^{l^2\xi^2}}{\xi^2} e^{i\xi(xy)}\ d^d\xi\ . $$ The RG is used in order to study quantities of the form $$ \int e^{V(\phi)}\ d\mu_{l,\infty}(\phi)\ . $$ The idea is to use a ``rescaling to unit lattice'', i.e., a scaling change of variable so one has an integral as before with $l=1$ (with a different $V$ that I will still call $V$ to keep notations simple). Then one uses a decomposition of Gaussian measures $$ \int e^{V(\phi)}\ d\mu_{1,\infty}(\phi) =\int \int e^{V(\psi+\zeta)} d\mu_{1,L}(\zeta)d\mu_{L,\infty}(\psi) $$ where $d\mu_{1,L}$ is the Gaussian measure corresponding to the covariance $C_{1,L}=C_{1,\infty}C_{L,\infty}$ and $L$ is some number $>1$. If one defines the constant $[\phi]=\frac{d2}{2}$, called the scaling dimension of the field, then the law of the field $\psi(x)$ is the same as that of $\phi_L(x)=L^{[\phi]}\phi(L^{1}x)$ where $\phi$ is sampled according to the original measure $d\mu_{1,\infty}$. Hence $$ \int e^{V(\phi)}\ d\mu_{1,\infty}(\phi) =\int \left(\int e^{V(\phi_L+\zeta)} d\mu_{1,L}(\zeta)\right)d\mu_{1,\infty}(\phi) $$ $$ =\int e^{V'(\phi)}\ d\mu_{1,\infty}(\phi) $$ where $$ V'(\phi)=\log\left( \int e^{V(\phi_L+\zeta)} d\mu_{1,L}(\zeta) \right)\ . $$ The renormalization group transformation on the space of Lagrangians is the map $V\rightarrow V'$. One can also do this infinitesimally by taking $L\rightarrow 1$, in which case one talks about an RG flow rather than a transformation. In the perturbative RG one writes the dynamical variable $V$ which is a complicated functional of the field as a formal power series in some variable which you can think of as Planck's constant. In the nonperturbative RG one essentially wants to use analysis to control the sum of this series. There are rigorous ways to study both RGs. The perturbative one is of course much simpler.
What you will find in Costello's book is only the perturbative RG. He does treat YangMills in flat space using the BatalinVilkovisky formalism, which is quite remarkable for an introductory book. For the curved case, see the paper http://arxiv.org/abs/0705.3340 by S. Hollands which appeared in J. Math. Phys.
If you would be happy learning about the RG flow on $\phi^4$ instead of YangMills, then much simpler perfectly rigorous presentations are available:
the book "Renormalization: an introduction" by Manfred Salmhofer, Springer, 1999.
the review article http://arxiv.org/abs/hepth/0208211 by Volkhard Mueller which appeared in Rev. Math. Phys.
As for the rigorous nonperturbative RG, the Park City lectures by Brydges mentioned by jc is definitely the best place to start. The issue here is that for Bosons one cannot really take the log in the definition of $V'$. This is called the large field problem, and one algebraic way around it is to use a socalled polymer representation. All this is explained by Brydges. Another nice introduction to the nonperturbative RG for Bosons is the set of lecture notes "Introduction to the Renormalization Group" by Antti Kupiainen.
For Fermions, taking the log is not a problem and good mathematical presentation can be found, e.g., in:
the book "Nonperturbative renormalization" by Vieri Mastropietro, World Sci. 2008.
the book "Renormalization group" by Giuseppe Benfatto and Giovanni Gallavotti, Princeton University Press, 1995.
the book "Fermionic Functional Integrals and the Renormalization Group" by Joel Feldman, Horst Knoerrer and Eugene Trubowitz, AMSCRM, 2002, also available at http://www.math.ubc.ca/~feldman/papers/aisenall.pdf
Also, for the nonperturbative RG there is another approach which is closer to BPHZ renormalization. It is presented in the book "From Perturbative to Constructive Renormalization" by Vincent Rivasseau.
If you would like a very short account of the kind of theorems one would like to prove in the nonperturbative RG setting you can also look up my recent Oberwolfach extended abstract: http://arxiv.org/abs/1104.2937
Edit: An in depth study of the rigorous nonperturbative RG is in the paper I just posted on arXiv: Rigorous quantum field theory functional integrals over the padics I: anomalous dimensions, with A. Chandra and G. Guadagni.
Update: A short pedagogical presentation, elaborating on the above answer can be found here. The slides of my recent talk "A Toy Model for ThreeDimensional Conformal Probability" also provides more details on how the renormalization group allows one to construct QFTs as weak limits of cutoff probability measures on spaces of Schwartz distributions, for the simplest example that contains the germs of generality. The precise formulation of the problem of constructing a QFT to be solved by renormalization group techniques can be found in my answer to this MO question.
Kevin Costello's book Renormalization and Effective Field Theory (Mathematical Surveys and Monographs) just came out. I've read a preliminary version of the text he was distributing earlier, and it is very good. It is by far the best mathematical treatment of renormalization from a pathintegral/Lagrangian pointofview that I know of.
A small complement to Abdelmalek Abdesselam's answer: on the rigorous, nonperturbative side, there is also a recent (originally twopart, now turned into threepart) exposition by Jonathan Dimock, available in the arXiv's. He uses the $\phi^4$ scalar field theory in 3 dimensions in finite volume as a model for his discussion  the three parts are listed below:
 J. Dimock, "The Renormalization Group According to Balaban I. Small Fields". arXiv:1108.1335 [mathph]
 J. Dimock, "The Renormalization Group According to Balaban II. Large Fields". arXiv:1212.5562 [mathph]
 J. Dimock, "The Renormalization Group According to Balaban III. Convergence". arXiv:1304.0705 [mathph]
Tadeusz Balaban refined the method of blockspin renormalization group employed by Gallavotti, Kupiainen and many others for lattice field models in order to analyse "large field" regions, aiming at the treatment of the continuum limit of pure YangMills models in finite volume and 4 dimensions. His long series of papers on the subject from the 80's remain essentially the state of the art towards the rigorous construction of realistic models in quantum field theory in 4 dimensions (see, for instance, the latest of the series), together with the paper of Magnen, Rivasseau and Sénéor, which was motivated by Balaban's work. The third part of Dimock's exposé is meant to establish the convergence of the expansion scheme laid down in Parts I and II.

1$\begingroup$ Yes these are good references too. $\endgroup$ Sep 9 '11 at 16:44
David C Brydges and his collaborators have been using renormalization inspired ideas to prove theorems about statistical mechanical systems for a few years now. From his page:
A large part of theoretical physics is built around the “functional integral” formulation of quantum field theory. These functional integrals are defined in the sense of formal power series (renormalised perturbation theory). It is widely, but wrongly believed, by mathematicians, that no precise definition that is useful for rigorous analysis is within sight. The renormalization group (RG), as pioneered by Ken Wilson (Nobel prize in Physics, 1982), provides a clear roadmap for defining functional integrals and studying the remarkable range of phenomena contained within them, in particular, renormalisation, scaling limits and the phase transitions of statistical mechanics. In these cases one can work with integrals based on measures on spaces of functions as opposed to complex valued "measures" on spaces of functions. The complex valued case (Feynman functional integrals) is indeed further toward the horizon of difficulty. Without facing the difficulties of the complex valued case, there is already an enormous range of possible applications. My interests in recent years have been in applications to selfavoiding walk in four dimensions. Functional integrals combine with supersymmetry to generate combinatoric identities so whenever I need a rest from the RG I like to think about that aspect as well. The papers below are a mixture of themes involving supersymmetry and analysis by RG. My colleague Joel Feldman is using closely related ideas to prove results in the context of condensed matter physics.
A good place to start reading about this line of research is his lecture notes from a 2007 PCMI summer conference on Statistical Mechanics. They are available for download from his webpage above, or in printed form as part of the proceedings from said conference.

$\begingroup$ And please see Abdelmalek Abdesselam's excellent answer for more context! $\endgroup$– j.c.Apr 26 '11 at 23:24
Langlands has thought about field theories for some time and has written a paper about renormalization group fixed points. The paper can be found among his collected works at his IAS website. The section on mathematical physics is located at this URL http://publications.ias.edu/rpl/section/28.

1$\begingroup$ @Laie: Thanks for pointing to that interesting paper! $\endgroup$ May 8 '11 at 7:13