What mathematical treatment is there on the renormalization group flow in a space of Lagrangians?

The answer to your question depends on whether you are interested in the perturbative RG or the nonperturbative one. Typically one starts with a Gaussian measure $d\mu_{0,\infty}$ on a space of fields $\phi$ given by a covariance $$C_{0,\infty}(x,y)=\int \phi(x)\phi(y)\ d\mu_{0,\infty}(\phi)\ . $$ For instance one can take for the covariance $\frac{1}{\xi^2}$ in Fourier space. Then one introduces a UV regularization at length scale $l$ by multiplying for instance by $\exp(l^2 \xi^2)$ which cutsoff momenta $\xi$ which are larger than $l^{1}$. This defines $$ C_{l,\infty}(x,y)=\frac{1}{(2\pi)^d}\int_{\mathbb{R}^d} \frac{e^{l^2\xi^2}}{\xi^2} e^{i\xi(xy)}\ d^d\xi\ . $$ The RG is used in order to study quantities of the form $$ \int e^{V(\phi)}\ d\mu_{l,\infty}(\phi)\ . $$ The idea is to use a ``rescaling to unit lattice'', i.e., a scaling change of variable so one has an integral as before with $l=1$ (with a different $V$ that I will still call $V$ to keep notations simple). Then one uses a decomposition of Gaussian measures $$ \int e^{V(\phi)}\ d\mu_{1,\infty}(\phi) =\int \int e^{V(\psi+\zeta)} d\mu_{1,L}(\zeta)d\mu_{L,\infty}(\psi) $$ where $d\mu_{1,L}$ is the Gaussian measure corresponding to the covariance $C_{1,L}=C_{1,\infty}C_{L,\infty}$ and $L$ is some number $>1$. If one defines the constant $[\phi]=\frac{d2}{2}$, called the scaling dimension of the field, then the law of the field $\psi(x)$ is the same as that of $\phi_L(x)=L^{[\phi]}\phi(L^{1}x)$ where $\phi$ is sampled according to the original measure $d\mu_{1,\infty}$. Hence $$ \int e^{V(\phi)}\ d\mu_{1,\infty}(\phi) =\int \left(\int e^{V(\phi_L+\zeta)} d\mu_{1,L}(\zeta)\right)d\mu_{1,\infty}(\phi) $$ $$ =\int e^{V'(\phi)}\ d\mu_{1,\infty}(\phi) $$ where $$ V'(\phi)=\log\left( \int e^{V(\phi_L+\zeta)} d\mu_{1,L}(\zeta) \right)\ . $$ The renormalization group transformation on the space of Lagrangians is the map $V\rightarrow V'$. One can also do this infinitesimally by taking $L\rightarrow 1$, in which case one talks about an RG flow rather than a transformation. In the perturbative RG one writes the dynamical variable $V$ which is a complicated functional of the field as a formal power series in some variable which you can think of as Planck's constant. In the nonperturbative RG one essentially wants to use analysis to control the sum of this series. There are rigorous ways to study both RGs. The perturbative one is of course much simpler. What you will find in Costello's book is only the perturbative RG. He does treat YangMills in flat space using the BatalinVilkovisky formalism, which is quite remarkable for an introductory book. For the curved case, see the paper http://arxiv.org/abs/0705.3340 by S. Hollands which appeared in J. Math. Phys. If you would be happy learning about the RG flow on $\phi^4$ instead of YangMills, then much simpler perfectly rigorous presentations are available:
As for the rigorous nonperturbative RG, the Park City lectures by Brydges mentioned by jc is definitely the best place to start. The issue here is that for Bosons one cannot really take the log in the definition of $V'$. This is called the large field problem, and one algebraic way around it is to use a socalled polymer representation. All this is explained by Brydges. Another nice introduction to the nonperturbative RG for Bosons is a set of (preliminary) lecture notes by Antti Kupiainen (you can find them on Google if you search for "Introduction to the renormalization group kupiainen"). For Fermions, taking the log is not a problem and good mathematical presentation can be found, e.g., in:
Also, for the nonperturbative RG there is another approach which is closer to BPHZ renormalization. It is presented in the book by Vincent Rivasseau http://www.cpht.polytechnique.fr/cpth/rivass/articles/book.ps If you would like a very short account of the kind of theorems one would like to prove in the nonperturbative RG setting you can also look up my recent Oberwolfach extended abstract: http://arxiv.org/abs/1104.2937 Edit: An in depth study of the rigorous nonperturbative RG is in the paper I just posted on arXiv: Rigorous quantum field theory functional integrals over the padics I: anomalous dimensions, with A. Chandra and G. Guadagni. Update (Nov 2013): A short pedagogical presentation, elaborating on the above answer can be found here. 


Kevin Costello's book Renormalization and Effective Field Theory (Mathematical Surveys and Monographs) just came out. I've read a preliminary version of the text he was distributing earlier, and it is very good. It is by far the best mathematical treatment of renormalization from a pathintegral/Lagrangian pointofview that I know of. 


A small complement to Abdelmalek Abdesselam's answer: on the rigorous, nonperturbative side, there is also a recent (originally twopart, now turned into threepart) exposition by Jonathan Dimock, available in the arXiv's. He uses the $\phi^4$ scalar field theory in 3 dimensions in finite volume as a model for his discussion  the three parts are listed below:
Tadeusz Balaban refined the method of blockspin renormalization group employed by Gallavotti, Kupiainen and many others for lattice field models in order to analyse "large field" regions, aiming at the treatment of the continuum limit of pure YangMills models in finite volume and 4 dimensions. His long series of papers on the subject from the 80's remain essentially the state of the art towards the rigorous construction of realistic models in quantum field theory in 4 dimensions (see, for instance, the latest of the series), together with the paper of Magnen, Rivasseau and Sénéor, which was motivated by Balaban's work. The third part of Dimock's exposé is meant to establish the convergence of the expansion scheme laid down in Parts I and II. 


David C Brydges and his collaborators have been using renormalization inspired ideas to prove theorems about statistical mechanical systems for a few years now. From his page:
A good place to start reading about this line of research is his lecture notes from a 2007 PCMI summer conference on Statistical Mechanics. They are available for download from his webpage above, or in printed form as part of the proceedings from said conference. 


Langlands has thought about field theories for some time and has written a paper about renormalization group fixed points. The paper can be found among his collected works at his IAS website. The section on mathematical physics is located at this URL http://publications.ias.edu/rpl/section/28. 

