You say, "There are physically impossible things that can be spoken about in mathematics." When mathematics is used to model a physical system, it is always an approximation. So, the fact that the model is not a perfect match for the system (that is, it describes some "physically impossible things") is not a surprise. There's no need to get fancy to show examples: our space is not plain old Euclidean 3 dimensional space, but that model has been very useful. Or even something as basic as possible: using natural numbers to count stuff. Well, it's physically possible that half of one of the things you were counting could burn up, and it turns out the mathematical model you were using is inadequate to describe the system (because you can't talk about half an object).
The fact that some mathematics is abstract but finds applications seems like a different issue. Some number theory is such an example, but I don't think anyone ever denied that integers were highly useful in modeling real systems (and thus, in some sense I guess, "real"); it's just that the questions people were asking about them seemed to not have practical uses.
Another issue is, just because a mathematical system is not a perfect model of the physical system you meant it to be, doesn't mean it isn't a better model of some other system. So you can't necessarily say a certain system is "physically impossible."
So I actually think there is less here than meets the eye. It's not really a big deal that we talk about infinite sets of things even though maybe there is no infinite collection of objects in reality, just like it's not really a big deal that we treat our space like it's continuous even though maybe it's actually discrete.
