Mathematicians and all the tools they can use are subject to the laws of physics which impose certain constraints, e.g. that you can only ever perform a finite number of elementary operations. In a universe governed by classical mechanics this is not true, there you could construct a machine whose clock cycle can increase exponentially fast. Such a machine could perform an infinite amount of computations in a finite time. It is because we don't live in such a universe that polynomials are useful to us.