As I'm sure you'll see from the many answers you'll get, there are lots of notions of "quantization". Here's another perspective.

Recall the primary motivation of, say, algebraic geometry: a geometric space is determined by its algebra of functions. Well, actually, this isn't quite true --- a complex manifold, for example, tends to have very few entire functions (any bounded entire function on C is constant, and so there are no nonconstant entire functions on a torus, say), so in algebraic geometry, they use "sheaves", which are a way of talking about local functions. In real geometry, though (e.g. topology, or differential geometry), there are partitions of unity, and it is more-or-less true that a space is determined by its algebra of total functions. Some examples: two smooth manifolds are diffeomorphic if and only if the algebras of smooth real-valued functions on them are isomorphic. Two locally compact Hausdorff spaces are homeomorphic if and only if their algebras of continuous real-valued functions that vanish at infinity (i.e. for any epsilon there is a compact set so that the function is less than epsilon outside the compact set) are isomorphic.

(From a physics point of view, it should be taken as a definition of "space" that it depends only on its algebra of functions. Said functions are the possible "observables" or "measurements" --- if you can't measure the difference between two systems, you have no right to treat them as different.)

So anyway, it can be useful to recast geometric ideas into algebraic language. Algebra is somehow more "finite" or "computable" than geometry.

But not every algebra arises as the algebra of functions on a geometric space. In particular, by definition the multiplication in the algebra is "pointwise multiplication", which is necessarily commutative (the functions are valued in R or C, usually).

So from this point of view, "quantum mathematics" is when you try to take geometric facts, written algebraically, and interpret them in a noncommutative algebra. For example, a space is locally compact Hausdorff iff its algebra of continuous functions is commutative c-star algebra, and any commutative c-star algebra is the algebra of continuous functions on some space (in fact, on its spectrum). So a "quantum locally compact Hausdorff space" is a non-commutative c-star algebra. Similarly, "quantum algebraic space" is a non-commutative polynomial algebra.

Anyway, I've explained "quantum", but not "quantization". That's because so far there's just geometry ("kinetics"), and no physics ("dynamics").

Well, a noncommutative algebra has, along with addition and multiplication, an important operation called the "commutator", defined by $[a,b]=ab-ba$. Noncommutativity says precisely that this operation is nontrivial. Let's pick a distinguished function H, and consider the operation $[H,-]$. This is necessarily a differential operator on the algebra, in the sense that it is linear and satisfies the Leibniz product rule. If the algebra were commutative, then differential operators would be the same as vector fields on the corresponding geometric space, and thus are the same as differential equations on the space. In fact, that's still true for noncommutative algebras: we define the "time evolution" by saying that for any function (=algebra element) f, it changes in time with differential [H,f]. (Using this rule on coordinate functions defines the geometric differential equation; in noncommutative land, there does not exist a complete set of coordinate functions, as any set of coordinate functions would define a commutative algebra.)

Ok, so it might happen that for the functions you care about, $[a,b]$ is very small. To make this mathematically precise, let's say that (for the subalgebra of functions that do not have very large values) there is some central algebra element $\hbar$, such that $[a,b]$ is always divisible by $\hbar$. Let $A$ be the algebra, and consider the $A/\hbar A$. If $\hbar$ is supposed to be a "very small number", then taking this quotient should only throw away fine-grained information, but some sort of "classical" geometry should still survive (notice that since $[a,b]$ is divisible by $\hbar$, it goes to $0$ in the quotient, so the quotient is commutative and corresponds to a classical geometric space). We can make this precise by demanding that there is a vector-space lift $(A/\hbar A) \to A$, and that $A$ is generated by the image of this lift along with the element $\hbar$.

Anyway, so with this whole set up, the quotient $A/\hbar A$ actually has a little more structure than just being a commutative algebra. In particular, since $[a,b]$ is divisible by $\hbar$, let's consider the element $\{a,b\} = \hbar^\{-1\} [a,b]$. (Let's suppose that $\hbar$ is not a zero-divisor, so that this element is well-defined.) Probably, $\{a,b\}$ is not small, because we have divided a small thing by a small thing, so that it does have a nonzero image in the quotient.

This defines on the quotient the structure of a Poisson algebra. In particular, you can check that $\{H,-\}$ is a differential operator for any (distinguished) element $H$, and so still defines a "mechanics", now on a classical space.

Then quantization is the process of reversing the above quotient. In particular, lots of spaces that we care about come with canonical Poisson structures. For example, for any manifold, the algebra of functions on its cotangent bundle has a Possion bracket. "Quantizing a manifold" normally means finding a noncommutative algebra so that some quotient (like the one above) gives the original algebra of functions on the cotangent bundle. The standard way to do this is to use Hilbert spaces and bounded operators, as I think another answerer described.