I would like to know what quantization is, I mean I would like to have some elementary examples, some soft nontechnical definition, some explanation about what do mathematicians quantize?, can we quantize a function?, a set?, a theorem?, a definition?, a theory?
11 Answers
As I'm sure you'll see from the many answers you'll get, there are lots of notions of "quantization". Here's another perspective.
Recall the primary motivation of, say, algebraic geometry: a geometric space is determined by its algebra of functions. Well, actually, this isn't quite true  a complex manifold, for example, tends to have very few entire functions (any bounded entire function on C is constant, and so there are no nonconstant entire functions on a torus, say), so in algebraic geometry, they use "sheaves", which are a way of talking about local functions. In real geometry, though (e.g. topology, or differential geometry), there are partitions of unity, and it is moreorless true that a space is determined by its algebra of total functions. Some examples: two smooth manifolds are diffeomorphic if and only if the algebras of smooth realvalued functions on them are isomorphic. Two locally compact Hausdorff spaces are homeomorphic if and only if their algebras of continuous realvalued functions that vanish at infinity (i.e. for any epsilon there is a compact set so that the function is less than epsilon outside the compact set) are isomorphic.
(From a physics point of view, it should be taken as a definition of "space" that it depends only on its algebra of functions. Said functions are the possible "observables" or "measurements"  if you can't measure the difference between two systems, you have no right to treat them as different.)
So anyway, it can be useful to recast geometric ideas into algebraic language. Algebra is somehow more "finite" or "computable" than geometry.
But not every algebra arises as the algebra of functions on a geometric space. In particular, by definition the multiplication in the algebra is "pointwise multiplication", which is necessarily commutative (the functions are valued in R or C, usually).
So from this point of view, "quantum mathematics" is when you try to take geometric facts, written algebraically, and interpret them in a noncommutative algebra. For example, a space is locally compact Hausdorff iff its algebra of continuous functions is commutative cstar algebra, and any commutative cstar algebra is the algebra of continuous functions on some space (in fact, on its spectrum). So a "quantum locally compact Hausdorff space" is a noncommutative cstar algebra. Similarly, "quantum algebraic space" is a noncommutative polynomial algebra.
Anyway, I've explained "quantum", but not "quantization". That's because so far there's just geometry ("kinetics"), and no physics ("dynamics").
Well, a noncommutative algebra has, along with addition and multiplication, an important operation called the "commutator", defined by $[a,b]=abba$. Noncommutativity says precisely that this operation is nontrivial. Let's pick a distinguished function H, and consider the operation $[H,]$. This is necessarily a differential operator on the algebra, in the sense that it is linear and satisfies the Leibniz product rule. If the algebra were commutative, then differential operators would be the same as vector fields on the corresponding geometric space, and thus are the same as differential equations on the space. In fact, that's still true for noncommutative algebras: we define the "time evolution" by saying that for any function (=algebra element) f, it changes in time with differential [H,f]. (Using this rule on coordinate functions defines the geometric differential equation; in noncommutative land, there does not exist a complete set of coordinate functions, as any set of coordinate functions would define a commutative algebra.)
Ok, so it might happen that for the functions you care about, $[a,b]$ is very small. To make this mathematically precise, let's say that (for the subalgebra of functions that do not have very large values) there is some central algebra element $\hbar$, such that $[a,b]$ is always divisible by $\hbar$. Let $A$ be the algebra, and consider the $A/\hbar A$. If $\hbar$ is supposed to be a "very small number", then taking this quotient should only throw away finegrained information, but some sort of "classical" geometry should still survive (notice that since $[a,b]$ is divisible by $\hbar$, it goes to $0$ in the quotient, so the quotient is commutative and corresponds to a classical geometric space). We can make this precise by demanding that there is a vectorspace lift $(A/\hbar A) \to A$, and that $A$ is generated by the image of this lift along with the element $\hbar$.
Anyway, so with this whole set up, the quotient $A/\hbar A$ actually has a little more structure than just being a commutative algebra. In particular, since $[a,b]$ is divisible by $\hbar$, let's consider the element $\{a,b\} = \hbar^{1} [a,b]$. (Let's suppose that $\hbar$ is not a zerodivisor, so that this element is welldefined.) Probably, $\{a,b\}$ is not small, because we have divided a small thing by a small thing, so that it does have a nonzero image in the quotient.
This defines on the quotient the structure of a Poisson algebra. In particular, you can check that $\{H,\}$ is a differential operator for any (distinguished) element $H$, and so still defines a "mechanics", now on a classical space.
Then quantization is the process of reversing the above quotient. In particular, lots of spaces that we care about come with canonical Poisson structures. For example, for any manifold, the algebra of functions on its cotangent bundle has a Possion bracket. "Quantizing a manifold" normally means finding a noncommutative algebra so that some quotient (like the one above) gives the original algebra of functions on the cotangent bundle. The standard way to do this is to use Hilbert spaces and bounded operators, as I think another answerer described.

2$\begingroup$ Concerning "...a space is locally compact Hausdorff iff its algebra of continuous functions is commutative cstar algebra": Unless I misunderstand the statement, it is not true: For any topological space $X$, the bounded functions $X\to \mathbb C$ form a commutative $C^*$algebra. $\endgroup$– RasmusCommented Nov 11, 2011 at 21:52

3$\begingroup$ @Rasmus: hrm, it's now been a while since cstaralgebra class, and it's not my area. But my understanding is the following. First, when I say "algebra of functions", I never mean the algebra of bounded functions. In the real world, I usually want "all" functions, but when I am working cstaralgebraically, I mean "function that's less than $\epsilon$ outside a compact". Given $X$, the algebra of bounded functions is the algebra of functions on the StoneCech completion $\beta X$ of $X$, and it's not surprising for $\beta X$ to have better properties than $X$. $\endgroup$ Commented Nov 12, 2011 at 6:41

3$\begingroup$ But of course you're right, there's something wrong with the statement, because any indiscrete space has only the constant functions, which clearly form a commutative cstar algebra. Probably I should have added the word "Hausdorff" somewhere  there's no chance of recovering nonHausdorff structure from continuous $\mathbb C$valued functions. $\endgroup$ Commented Nov 12, 2011 at 6:43
I don't know what it means for a mathematician to quantize something, but I can give you a rough description, and a few specific examples, from a physicist's point of view.
Motivational fluff
When quantum mechanics was first discovered, people tended to think of it as a modified version of classical mechanics [1]. In those days, very few quantum systems were known, so people would create quantum systems by "quantizing" classical ones. To quantize a classical system is to come up with a quantum system that "behaves similarly" in some sense. For example, you generally want there to be an intuitive correspondence between the observables of a classical system and the observables of its quantization, and you generally want the expectation values of the quantized observables to obey the same equations of motion as their classical counterparts.
Because the goal of quantization is to find a quantum system that's "analogous" in some way to a given classical system, it's not a mathematically welldefined procedure, and there's no unique way of doing it. How you attempt to quantize a system, and how you decide whether or not you've succeeded, depends entirely on your motivation and goals.
The harder stuff
I've been using the phrase "quantum system" a lotwhat do I really mean? In my opinion, one of the best ways to find out is to read Section 16.5 of Probability via Expectation, by Peter Whittle.
Roughly speaking, a quantum system has two basic parts:
A complex inner product space $H$, called the state space [2]. Each ray of $H$ represents a possible "pure state" of the system. A pure state is somewhat analogous to a probability distribution, in that it tells you how to assign expectation values to "observables"; in particular, it tells you how to assign probabilities to propositions.
A collection of selfadjoint linear maps from $H$ to itself, called observables. An observable is somewhat analogous to a random variable; it represents a property of the system that can be measured and found to have a certain value. The values that an observable can take are given by its eigenvalues (or, in the infinitedimensional case, its spectrum). Say $A$ is an observable, $a$ is an eigenvalue of $A$, and $v_1, \ldots, v_n \in H$ form an orthonormal basis for the eigenspace of $a$. If the state of the system is the ray generated by the unit vector $\psi \in H$, the probability that the observable $A$ will be found to have the value $a$ is $\langle v_1, \psi \rangle + \ldots + \langle v_n, \psi \rangle$, where $\langle \cdot, \cdot \rangle$ is the inner product. You can then easily show that the expectation value of the observable $A$ is $\langle \psi A \psi \rangle$. Observables whose only eigenvalues are $1$ and $0$—that is, projection operators on $H$—play a special role, because they correspond to logical propositions about the system. The expectation value of a projection operator is just the probability of the proposition.
Most interesting quantum systems have another part, which is often very important:
 A set of unitary maps from $H$ to itself, which might be called transformations. These represent "automorphisms" of the system. In physics, many quantum systems have a oneparameter group of transformations, often denoted $U(t)$, that represent time evolution; the idea is that if the state of the system is currently (the ray generated by) $\psi$, the state will be $U(t)\psi$ after $t$ units of time have passed. Physical systems often have other transformation groups as well; for example, a quantum system that's supposed to have a "spatial orientation" will generally have a group of transformations that form a representation of $SO(3)$.
A few examples
Quantum random walks are, as the name suggests, quantized random walks. More generally, you can quantize the idea of a Markov chain. For a great introduction, see the paper "Quantum walks and their algorithmic applications", by Andris Ambainis.
In Sections 2 and 3 of the notes "A Short Introduction to Noncommutative Geometry", Peter Bongaarts describes quantized versions of compact topological spaces and classical mechanical systems.
In Section 4 of the book Noncommutative Geometry (cautionbig PDF), Alain Connes introduces a quantized version of calculus. Here, the observables representing complex variables are nonselfadjoint because complex variables can take on complex values. An observable representing a complex variable must therefore be allowed to have complex eigenvalues.
I hope this helps!
[1] Today, in contrast, most physicists think of classical mechanics as an approximation to quantum mechanics.
[2] If $H$ is infinitedimensional, it's typically a separable Hilbert space. You may even need $H$ to be something fancier, like a rigged Hilbert space.

$\begingroup$ This is very old, but shouldn't you be using the modulus squared of the inner product for the probability of measuring the system in the $a$ eigenspace? $\endgroup$– EE18Commented Apr 23, 2022 at 14:41
Just to restate some facts already stated in other answers, quantization can mean a few different things. In deformation quantization, we start with a classical theory given by a Poisson manifold. Then, (by definition) the algebra of functions forms a Poisson algebra. A quantization of this algebra is a noncommutative algebra with operators $X_f$ for $f$ a function. There is also a formal parameter $\hbar$. This algebra satifies $$ X_f\ X_g = X_{fg} + \mathcal{O}(\hbar)\ . $$
The idea of quantization is that the Poisson bracket becomes a commutator, or
$$ [X_f,X_g] = \hbar X_{\lbrace f,g \rbrace} + \mathcal{O}(\hbar^2)\ . $$
Thus, we have a noncommutative version of classical mechanics. The existence of such an algebra is a theorem of Kontsevich (the case of a symplectic manifold was solved much earlier, but I forget by whom).
In mathematics, there are plenty of interesting analogous situations where you have a noncommutative thingie which is, in some sense, a formal deformation of a commutative thingie. You can see the other direction of the above as an example of the following general fact. Given a filtered algebra whose associated graded is commutative, there is a natural Poisson structure on the associated graded.
In physics, however, it's not enough to just deform the algebra of functions; we have to now represent things on a Hilbert space. This introduces a whole host of other problems. In geometric quantization, this is split into two steps. Let's say we have a symplectic manifold whose symplectic form is integral. Then we can construct a line bundle with connection whose curvature is that symplectic form. The Hilbert space is the space of $L^2$ sections of this bundle. This is much too large, however, so you have to cut it down (which is step 2). In various cases, welldefined procedures exist, but I don't believe this is wellunderstood in general. For example, I'm not sure it's possible to represent every function as an operator.
It's probably worth pointing out that, from the point of view of physics, quantization is backwards. It is the quantum theory that is fundamental, and the classical theory should arise as some limit of the quantum theory. There's some interesting mathematics there, and also a whole lot of philosophy too.

2$\begingroup$ I believe that the symplectic case was solved independently by De WildeLecomte, OmoriMaedaYoshioka and Fedosov. $\endgroup$ Commented Nov 20, 2009 at 17:43
There are some good long answers already, so I'm going to try to give as short an answer as possible.
A quantization of $X$ is some $X_\hbar$ depending on a parameter $\hbar$ (occasionally $q=e^\hbar$ instead) such that $X=X_0$ and $X_\hbar$ is generically "less commutative" than $X$. This is by analogy with quantum physics where $X_0$ is classical physics and $\hbar$ measures the failure of position and momentum to commute.
The word has many meanings in mathematics, most of them quite vague.
One general way of describing what quantization is for a mathematician is the following: you have your favorite object $X$, and you find that there is a family of other objects $X_q$ parametrized by a parameter $q$ which varies in some set (or is only a â€˜formal parameterâ€™ in the way that the variable in a polynomial ring is â€˜formallyâ€™ an element in a overring of the coefficient ring) such that for a special value $q_0$ of the parameter $q$, or, in the â€˜formalâ€™ case, when the parameter degenerates in some specific way, you have that $X_{q_0}$ is your original favorite $X$, and if the objects $X_q$ are in some sense (more) noncommutative than $X$, one says that the family $X_q$ is a quantization of $X$.
Very vague, I know. And this is only interesting if both your $X$ is interesting, if the $X_q$ themselves are interesting, and if there is some connection between the two.
For example, integer numbers are undeniably interesting objects, and they have a â€˜quantizationâ€™, given by the (one of the couple of) usual quantum integers where this is very visible.
The thing is, usually, starting from some interesting $X$, there are really not very many ways in which you can do this. For example, if you start with an enveloging algebra of a simple Lie algebra over $\mathbb C$, then there is just one way to do this (up to the appropriate way of ignoring that there are really many ways to do this)

$\begingroup$ I think you mean quantization is some kind of deformation theory. $\endgroup$– AllenCommented Sep 28, 2012 at 11:23
In mathematics, quantization often refers to some kind of deformation of a classical object. The Heisenberg Uncertainty Principle says that the position and momentum operators do not commute. In fact, $[X,P]=i\hbar$. In the limit as $\hbar\to 0$, these operators commute once again. Technically speaking, this is nonsense as $\hbar$ is a universal constant, but in mathematics, we are free to play with parameters. A couple of examples include:
 the noncommutative torus, the universal $C^\ast$algebra generated by two unitaries satisfying $uv=e^{i\theta} vu$. As $\theta\to 0$, we get $C(\mathbb{T}^2)$, the continuous functions on the $2$torus. We usually think of the deformed algebra as a quantization of the commutative one.
 some quantum groups are deformations of universal enveloping algebras, i.e., we get the universal enveloping algebra as $q\to 1$.
A very basic answer: think about the classical Hamiltonian, $$ a(x,\xi)=\vert \xi\vert^2\frac{\kappa}{\vert x\vert},\quad \text{$\kappa>0$ parameter}. $$ The classical motion is described by the integral curves of the Hamiltonian vector field of $a$, $$ \dot x=\frac{\partial a}{\partial\xi},\quad \dot \xi=\frac{\partial a}{\partial x}. $$ The attempt of describing the motion of an electron around a proton by classical mechanics leads to the study of the previous integral curves and is extremely unstable since the function $a$ is unbounded from below. If classical mechanics were governing atomic motion, matter would not exist, or would be so unstable that could not sustain its observed structure for a long time, with electrons collapsing onto the nucleus.
Now, you change the perspective and you decide, quite arbitrarily that atomic motion will be governed by the spectral theory of the quantization of $a$, i.e. by the selfadjoint operator $$ \Delta\frac{\kappa}{\vert x\vert}=A. $$ It turns out that the spectrum of that operator is bounded from below by some fixed negative constant, and this a way to explain stability of matter. Moreover the eigenvalues of $A$ are describing with an astonishing accuracy the levels of energy of an electron around a proton (hydrogen atom).
My point is that, although quantization has many various mathematical interpretations, its success is linked to a striking physical phenomenon: matter is existing with some stability and no explanation of that fact has a classical mechanics interpretation. The atomic mechanics should be revisited, and quantization is quite surprisingly providing a rather satisfactory answer. For physicists, it remains a violence that so refined mathematical objects (unbounded operators acting on  necessarily  infinite dimensional space) have so many things to say about nature. It's not only Einstein's "God does not play dice", but also Feynman's "Nobody understands Quantum Mechanics" or Wigner's "Unreasonable effectiveness of Mathematics."

1$\begingroup$ I vote for "Nobody understands Quantum Mechanics". You are not Joking, Mr. Feynman :) $\endgroup$ Commented Nov 20, 2013 at 1:10
As a physicist who has taken a bunch of Quantum Mechanics and Solid State physics, when we say "quantize your system" it means:
You set up your classical Lagrangian $L$ (in terms of kinetic $K$ and potential $U$ energy), given generalized coordinates $q_i,p_i$ (usually position and momentum, but could also be angles and angular velocity). You then take the Hamiltonian $H$ of that system, which in most cases becomes $H=K+U$. This is all in terms of your generalized coordinates.
Once that is done, "quantizing" the system (or your variables) means to simply set $[q_i,p_j]=i\hbar \delta_{ij}$. The quantum mechanics is now in effect. This is known as $\textit{canonical quantization}$.
Quantum Field Theory is a perturbation to Quantum Mechanics, where you perform a second quantization. For instance, in using electrodynamics in quantum mechanics you simply quantize the atomicmotion (which interacts with the $\textbf{E}$field); this is the "semiclassical approach". Second quantization further quantizes this electromagnetic field, so that now the light and the atom both have discrete structures.
Given a theory, described by an action $S(\phi)$ with field $\phi \in \mathcal{P}$, where $\mathcal{P}$ is usually the set of sections of a bundle over some manifold $M$. The action admits $\mathcal{G}$ a set of gauge symmetries, $\phi \rightarrow \phi'$ such that $S(\phi) = S(\phi')$.
One has quantized this theory when one has calculated, or has an algorithm that can calculate
$\int_{\mathcal{P} / \mathcal{G}} \mathcal{O}(\phi) e^{iS(\phi)/\hbar} \mathcal{D}\phi$
for any function $\mathcal{O}(\phi)$ on $\mathcal{P} / \mathcal{G}$.
In the case of quantum field theory $\mathcal{D}\phi$ is usually illdefined and the integral usually diverges. However, for a certain class of theories, socalled renormalizable theories, one can, moreorless, make sense of this integral.
An excellent treatment of perturbative renormalization, from a mathematical pointofview, is found in Kevin Costello's soon to be published book, Renormalization and effective field theory.
I'm gonna be a bit more down to earth and cover the basics of Weyl quantization (in units where $\hbar = 1$)...
The Hamiltonian is typically introduced first: starting from the de Broglie relation $p = k$ and the EinsteinPlanck relation $E = \omega$ we can regard the (Weyl) correspondence principle heuristically as arising by viewing Fourier analysis through the lens of spectral theory for selfadjoint operators: i.e., we have
$p \rightarrow i\partial_x, \quad H \rightarrow i\partial_t$
which leads immediately to the SchrÃ¶dinger equation, in which the energy levels are associated with eigenvalues of the Hamiltonian. The Euclidean version is obtained by a Wick rotation:
$t = i\tau \Rightarrow \partial_t = \partial_{i\tau} = i\partial_\tau \Rightarrow H \rightarrow \partial_\tau.$
The time evolution operator encoding the dynamics is just $U(t) = e^{iHt}$. The rest is details or field theory.
Here is a link to an article on quantization in physics:
http://en.wikipedia.org/wiki/Quantization_(physics)
The article contains links to other articles on quantization including cananonical quantization and geometric quantization, and weyl quantization. quantization involves converting classical fields to operators acting on quantum states of the field theory.