2 Fixed mistakes in observable section, added stuff about projections
• A collection of self-adjoint linear maps from $H$ to itself, called observables. An observable is somewhat analogous to a random variable; it represents a property of the system that can be measured and found to have a certain value. The values that an observable can take are given by its eigenvalues (or, in the infinite-dimensional case, its spectrum). Say $A$ is an observable, and $v$ a$is the unique eigenvector an eigenvalue of$A$with eigenvalue A$, and $v_1, \ldots, v_n \in H$ form an orthonormal basis for the eigenspace of $a$. If the state of the system is the ray generated by the unit vector $\psi \in H$, the probability that the observable $A$ will be found to have the value $a$ is $\langle v_1, \psi, A\psi psi \rangle + \ldots + \langle v_n, \psi \rangle$, where $\langle \cdot, \cdot \rangle$ is the inner product. You can then easily show that the expectation value of the observable $A$ is $\langle \psi A \psi \rangle$. Observables whose only eigenvalues are $1$ and $0$—that is, projection operators on $H$—play a special role, because they correspond to logical propositions about the system. The expectation value of a projection operator is just the probability of the proposition.

• A set of unitary maps from $H$ to itself, which might be called transformations. These represent "automorphisms" of the system. In physics, many quantum systems have a one-parameter group of transformations, often denoted $U(t)$, that represent time evolution; the idea is that if the state of the system is currently (the ray generated by) $\psi$, the state will be $U(t)\psi$ after $t$ units of time have passed. Physical systems often have other transformation groups as well; for example, a quantum system that's supposed to have a "spatial orientation" will generally have a group of transformations that form a representation of $SO(3)$.
• 1 [made Community Wiki]

I don't know what it means for a mathematician to quantize something, but I can give you a rough description, and a few specific examples, from a physicist's point of view.

Motivational fluff

When quantum mechanics was first discovered, people tended to think of it as a modified version of classical mechanics [1]. In those days, very few quantum systems were known, so people would create quantum systems by "quantizing" classical ones. To quantize a classical system is to come up with a quantum system that "behaves similarly" in some sense. For example, you generally want there to be an intuitive correspondence between the observables of a classical system and the observables of its quantization, and you generally want the expectation values of the quantized observables to obey the same equations of motion as their classical counterparts.

Because the goal of quantization is to find a quantum system that's "analogous" in some way to a given classical system, it's not a mathematically well-defined procedure, and there's no unique way of doing it. How you attempt to quantize a system, and how you decide whether or not you've succeeded, depends entirely on your motivation and goals.

The harder stuff

I've been using the phrase "quantum system" a lot---what do I really mean? In my opinion, one of the best ways to find out is to read Section 16.5 of Probability via Expectation, by Peter Whittle.

Roughly speaking, a quantum system has two basic parts:

• A complex inner product space $H$, called the state space [2]. Each ray of $H$ represents a possible "pure state" of the system. A pure state is somewhat analogous to a probability distribution, in that it tells you how to assign expectation values to "observables"; in particular, it tells you how to assign probabilities to propositions.

• A collection of self-adjoint linear maps from $H$ to itself, called observables. An observable is somewhat analogous to a random variable; it represents a property of the system that can be measured and found to have a certain value. The values that an observable can take are given by its eigenvalues (or, in the infinite-dimensional case, its spectrum). Say $A$ is an observable, and $v$ is the unique eigenvector of $A$ with eigenvalue $a$. If the state of the system is $\psi \in H$, the probability that the observable $A$ will be found to have the value $a$ is $\langle \psi, A\psi \rangle$, where $\langle \cdot, \cdot \rangle$ is the inner product.

Most interesting quantum systems have another part, which is often very important:

• A set of unitary maps from $H$ to itself, which might be called transformations. These represent "automorphisms" of the system. In physics, many quantum systems have a one-parameter group of transformations, often denoted $U(t)$, that represent time evolution; the idea is that if the state of the system is currently $\psi$, the state will be $U(t)\psi$ after $t$ units of time have passed. Physical systems often have other transformation groups as well; for example, a quantum system that's supposed to have a "spatial orientation" will generally have a group of transformations that form a representation of $SO(3)$.

A few examples

• Quantum random walks are, as the name suggests, quantized random walks. More generally, you can quantize the idea of a Markov chain. For a great introduction, see the paper "Quantum walks and their algorithmic applications", by Andris Ambainis.

• In Sections 2 and 3 of the notes "A Short Introduction to Noncommutative Geometry", Peter Bongaarts describes quantized versions of compact topological spaces and classical mechanical systems.

• In Section 4 of the book Noncommutative Geometry (caution---big PDF), Alain Connes introduces a quantized version of calculus. Here, the observables representing complex variables are non-self-adjoint because complex variables can take on complex values. An observable representing a complex variable must therefore be allowed to have complex eigenvalues.

I hope this helps!

[1] Today, in contrast, most physicists think of classical mechanics as an approximation to quantum mechanics.

[2] If $H$ is infinite-dimensional, it's typically a separable Hilbert space. You may even need $H$ to be something fancier, like a rigged Hilbert space.