A collection of self-adjoint linear maps from $H$ to itself, called observables. An observable is somewhat analogous to a random variable; it represents a property of the system that can be measured and found to have a certain value. The values that an observable can take are given by its eigenvalues (or, in the infinite-dimensional case, its spectrum). Say $A$ is an observable, and $v$ a$ is the unique eigenvector an eigenvalue of $A$ with eigenvalue A$, and $v_1, \ldots, v_n \in H$ form an orthonormal basis for the eigenspace of $a$. If the state of the system is the ray generated by the unit vector $\psi \in H$, the probability that the observable $A$ will be found to have the value $a$ is $\langle v_1, \psi, A\psi psi \rangle + \ldots + \langle v_n, \psi \rangle$, where $\langle \cdot, \cdot \rangle$ is the inner product. You can then easily show that the expectation value of the observable $A$ is $\langle \psi A \psi \rangle$. Observables whose only eigenvalues are $1$ and $0$—that is, projection operators on $H$—play a special role, because they correspond to logical propositions about the system. The expectation value of a projection operator is just the probability of the proposition.
|2||Fixed mistakes in observable section, added stuff about projections|
|1||[made Community Wiki]|
I don't know what it means for a mathematician to quantize something, but I can give you a rough description, and a few specific examples, from a physicist's point of view.
When quantum mechanics was first discovered, people tended to think of it as a modified version of classical mechanics . In those days, very few quantum systems were known, so people would create quantum systems by "quantizing" classical ones. To quantize a classical system is to come up with a quantum system that "behaves similarly" in some sense. For example, you generally want there to be an intuitive correspondence between the observables of a classical system and the observables of its quantization, and you generally want the expectation values of the quantized observables to obey the same equations of motion as their classical counterparts.
Because the goal of quantization is to find a quantum system that's "analogous" in some way to a given classical system, it's not a mathematically well-defined procedure, and there's no unique way of doing it. How you attempt to quantize a system, and how you decide whether or not you've succeeded, depends entirely on your motivation and goals.
The harder stuff
I've been using the phrase "quantum system" a lot---what do I really mean? In my opinion, one of the best ways to find out is to read Section 16.5 of Probability via Expectation, by Peter Whittle.
Roughly speaking, a quantum system has two basic parts:
Most interesting quantum systems have another part, which is often very important:
A few examples
I hope this helps!
 Today, in contrast, most physicists think of classical mechanics as an approximation to quantum mechanics.
 If $H$ is infinite-dimensional, it's typically a separable Hilbert space. You may even need $H$ to be something fancier, like a rigged Hilbert space.