Joël asked in a comment that amplified on the original question: "What is not clear to me is how this hypothesis get translated in the quantum setting... I was told that the corresponding quantum system is the space V of function on B vanishing on δB, and the Hamiltonian is the Laplacian. But how does that translate the hypothesis about bouncing? If we assume a weird bouncing where the the inward angle is said twice the outward angle, what would be the attached quantum system?"
A basic requirement for any method of quantization is that it should recover the classical behavior in the limit $h\rightarrow0$. As a first example, let's take the standard classical billiard-ball system, with equal angles of incidence and reflection $\theta_r=\theta_i$, and do our quantization using the path integral method. In the limit $h\rightarrow0$, you get complete interference for all paths except for the one of extremal length, which is the one that has equal angles of incidence and reflection.
Now let's consider your system with $\theta_r=\theta_i/2$. The time-evolution of this system doesn't preserve volume in phase space, so by Liouville's theorem it can't be described by a Hamiltonian. Typically a quantization method starts from a Hamiltonian description, so that's a problem. Physically, the rule $\theta_r=\theta_i/2$ lacks time-reversal symmetry, so under quantization, I suppose the wave equation would have to have a first derivative with respect to time in it. I think what this example shows is that quantization methods are like a set of tools that are made to be used for different purposes. Only certain kinds of classical systems have quantized counterparts that are of interest, and each quantization method is a tool that was only designed to be used on certain classes of systems of interest. As another example, the quantization methods used for the electromagnetic field fail when applied to gravitational fields.
Other answers have discussed the fact that quantization is not a turn-key process. To make this more concrete, I think it's helpful to consider the simplest example I know that is of actual physical interest, which is an electron in an externally applied electromagnetic field. There you get the Aharonov-Bohm effect, which is a nontrivial nonclassical effect that would be hard to anticipate.
