Given a theory, described by an action $S(\phi)$ with field $\phi \in \mathcal{P}$, where $\mathcal{P}$ is usually the set of sections of a bundle over some manifold $M$. The theory action admits $\mathcal{G}$ a set of gauge symmetries, $\phi \rightarrow \phi'$ such that $S(\phi) = S(\phi')$.
One has quantized this theory when one has calculated, or has an algorithm that can calculate
$\int_{\mathcal{P} / \mathcal{G}} \mathcal{O}(\phi) e^{iS(\phi)/\hbar} \mathcal{D}\phi$
for any function $\mathcal{O}(\phi)$ on $\mathcal{P} / \mathcal{G}$.
In the case of quantum field theory $\mathcal{D}\phi$ is usually ill-defined and the integral usually diverges. However, for a certain class of theories, so-called renormalizable theories, one can, more-or-less, make sense of this integral.
An excellent treatment of perturbative renormalization, from a mathematical point-of-view, is found in Kevin Costello's soon to be published book, Renormalization and effective field theory.
