What mathematical treatment is there on the renormalization group flow in a space of Lagrangians? What mathematical treatment is there on the renormalization group flow in a space of Lagrangians?
 A: David C Brydges and his collaborators have been using renormalization inspired ideas to prove theorems about statistical mechanical systems for a few years now.  From his page: 

A large part of theoretical physics is built around the “functional integral” formulation of quantum field theory.  These functional integrals are defined in the sense of formal power series (renormalised perturbation theory).  It is widely, but wrongly believed, by mathematicians, that no precise definition that is useful for rigorous analysis is within sight.  The renormalization group (RG), as pioneered by Ken Wilson (Nobel prize in Physics, 1982),  provides a clear roadmap for defining functional integrals and studying the remarkable  range of phenomena contained within them, in particular, renormalisation, scaling limits and the phase transitions of statistical mechanics. In these cases one can work with integrals based on measures on spaces of functions as opposed to complex valued "measures" on spaces of functions.  The complex valued case (Feynman functional integrals) is indeed further toward the horizon of difficulty.  Without facing the difficulties of the complex valued case, there is already an enormous range of possible applications.  My interests in  recent years have been in applications to  self-avoiding walk in four dimensions.  Functional integrals  combine with supersymmetry to generate combinatoric identities so whenever I need a rest from the RG I like to think about that aspect as well. The papers below are a mixture of themes involving supersymmetry and analysis by RG.    My colleague Joel Feldman is using closely related ideas to prove results in the context of condensed matter physics. 

A good place to start reading about this line of research is his lecture notes from a 2007 PCMI summer conference on Statistical Mechanics.  They are available for download from his webpage above, or in printed form as part of the proceedings from said conference.  
A: Langlands has thought about field theories for some time and has written a paper about  renormalization group fixed points. The paper can be found among his collected works at his IAS website. The section on mathematical physics is located at this URL
http://publications.ias.edu/rpl/section/28.
A: Dec 2017 edit: I just put a hopefully useful detailed complement to this post on https://physics.stackexchange.com/questions/372306/wilsonian-definition-of-renormalizability

The answer to your question depends on whether you are interested in the perturbative RG
or the nonperturbative one.
Typically one starts with a Gaussian measure $d\mu_{0,\infty}$ on a space of fields $\phi$
given by a covariance
$$C_{0,\infty}(x,y)=\int \phi(x)\phi(y)\ d\mu_{0,\infty}(\phi)\ .
$$
For instance one can take for the covariance $\frac{1}{\xi^2}$ in Fourier space.
Then one introduces a UV regularization at length scale $l$ by multiplying
for instance by $\exp(-l^2 \xi^2)$ which cuts-off momenta $\xi$ which are larger than $l^{-1}$. This defines
$$
C_{l,\infty}(x,y)=\frac{1}{(2\pi)^d}\int_{\mathbb{R}^d}
\frac{e^{-l^2\xi^2}}{\xi^2} e^{i\xi(x-y)}\ d^d\xi\ .
$$
The RG is used in order to study quantities of the form
$$
\int e^{-V(\phi)}\ d\mu_{l,\infty}(\phi)\ .
$$
The idea is to use a ``rescaling to unit lattice'', i.e., a scaling change of variable
so one has an integral as before with $l=1$ (with a different $V$ that I will still call
$V$ to keep notations simple).
Then one uses a decomposition of Gaussian measures
$$
\int e^{-V(\phi)}\ d\mu_{1,\infty}(\phi)
=\int \int e^{-V(\psi+\zeta)} d\mu_{1,L}(\zeta)d\mu_{L,\infty}(\psi)
$$
where $d\mu_{1,L}$ is the Gaussian measure corresponding to the covariance
$C_{1,L}=C_{1,\infty}-C_{L,\infty}$ and $L$ is some number $>1$.
If one defines the constant $[\phi]=\frac{d-2}{2}$, called the scaling dimension
of the field, then
the law of the field $\psi(x)$ is the same as
that of $\phi_L(x)=L^{-[\phi]}\phi(L^{-1}x)$
where $\phi$ is sampled according to the original measure
$d\mu_{1,\infty}$. Hence
$$
\int e^{-V(\phi)}\ d\mu_{1,\infty}(\phi)
=\int \left(\int e^{-V(\phi_L+\zeta)} d\mu_{1,L}(\zeta)\right)d\mu_{1,\infty}(\phi)
$$
$$
=\int e^{-V'(\phi)}\ d\mu_{1,\infty}(\phi)
$$
where
$$
V'(\phi)=-\log\left(
\int e^{-V(\phi_L+\zeta)} d\mu_{1,L}(\zeta)
\right)\ .
$$
The renormalization group transformation on the space of Lagrangians
is the map $V\rightarrow V'$.
One can also do this infinitesimally by taking $L\rightarrow 1$, in which case
one talks about an RG flow rather than a transformation.
In the perturbative RG one writes the dynamical variable $V$ which is a complicated
functional of the field as a formal power series in some variable which you can think of as Planck's constant.
In the nonperturbative RG one essentially wants to use analysis to control the sum of this series. There are rigorous ways to study both RGs. The perturbative one is of course
much simpler.
What you will find in Costello's book is only the perturbative RG. He does treat
Yang-Mills in flat space using the Batalin-Vilkovisky formalism, which is quite
remarkable for an introductory
book. For the curved case, see the paper 
http://arxiv.org/abs/0705.3340
by S. Hollands which appeared in J. Math. Phys.
If you would be happy learning about the RG flow on $\phi^4$ instead of Yang-Mills,
then much simpler perfectly rigorous presentations are available:


*

*the book "Renormalization: an introduction" by Manfred Salmhofer, Springer, 1999.

*the review article http://arxiv.org/abs/hep-th/0208211  by Volkhard Mueller which
appeared in Rev. Math. Phys.
As for the rigorous nonperturbative RG, the Park City lectures by Brydges mentioned by jc is definitely the best place to start.
The issue here is that for Bosons one cannot really take the log in the definition
of $V'$. This is called the large field problem, and one algebraic way around
it is to use a so-called polymer representation. All this is explained by Brydges.
Another nice introduction to the nonperturbative RG for Bosons is the set of lecture notes "Introduction to the Renormalization Group" by Antti Kupiainen.
For Fermions,
taking the log is not a problem and good mathematical presentation can be found, e.g.,
in:


*

*the book "Non-perturbative renormalization" by Vieri Mastropietro, World Sci. 2008.

*the book "Renormalization group" by Giuseppe Benfatto and Giovanni Gallavotti, Princeton University Press, 1995.

*the book "Fermionic Functional Integrals and the Renormalization Group"
by Joel Feldman, Horst Knoerrer and Eugene Trubowitz, AMS-CRM, 2002,
also available at http://www.math.ubc.ca/~feldman/papers/aisen-all.pdf
Also, for the nonperturbative RG there is another approach which is closer to BPHZ
renormalization. It is presented in the book "From Perturbative to Constructive Renormalization" by Vincent Rivasseau.
If you would like a very short account of the kind of theorems one would like to prove
in the nonperturbative RG setting you can also look up my recent Oberwolfach extended abstract: http://arxiv.org/abs/1104.2937

Edit: An in depth study of the rigorous nonperturbative RG
is in the paper I just posted on arXiv:
Rigorous quantum field theory functional integrals over the p-adics I: anomalous dimensions, with A. Chandra and G. Guadagni.

Update: A short pedagogical presentation, elaborating on the above answer can be found here. The slides of my recent talk "A Toy Model for Three-Dimensional Conformal Probability" also provides more details on how the renormalization group allows one to construct QFTs as weak limits of cut-off probability measures on spaces of Schwartz distributions, for the simplest example that contains the germs of generality. The precise formulation of the problem of constructing a QFT to be solved by renormalization group techniques can be found in my answer to this MO question.
A: Kevin Costello's book Renormalization and Effective Field Theory (Mathematical Surveys and Monographs) just came out. I've read a preliminary version of the text he was distributing earlier, and it is very good. It is by far the best mathematical treatment of renormalization from a path-integral/Lagrangian point-of-view that I know of. 
A: A small complement to Abdelmalek Abdesselam's answer: on the rigorous, non-perturbative side, there is also a recent (originally two-part, now turned into three-part) exposition by Jonathan Dimock, available in the arXiv's. He uses the $\phi^4$ scalar field theory in 3 dimensions in finite volume as a model for his discussion - the three parts are listed below:

*

*J. Dimock, "The Renormalization Group According to Balaban I. Small Fields". arXiv:1108.1335 [math-ph]

*J. Dimock, "The Renormalization Group According to Balaban II. Large Fields". arXiv:1212.5562 [math-ph]

*J. Dimock, "The Renormalization Group According to Balaban III. Convergence". arXiv:1304.0705 [math-ph]

Tadeusz Balaban refined the method of block-spin renormalization group employed by Gallavotti, Kupiainen and many others for lattice field models in order to analyse "large field" regions, aiming at the treatment of the continuum limit of pure Yang-Mills models in finite volume and 4 dimensions. His long series of papers on the subject from the 80's remain essentially the state of the art towards the rigorous construction of realistic models in quantum field theory in 4 dimensions (see, for instance, the latest of the series), together with the paper of Magnen, Rivasseau and Sénéor, which was motivated by Balaban's work. The third part of Dimock's exposé is meant to establish the convergence of the expansion scheme laid down in Parts I and II.
