I agree with Mike that even the example of $\mathbb{R}^3$ shows that talking about infinite mathematical objects can still make finite physical sense. For example, in $\mathbb{R}^3$ we can show that there are only so many regular polyhedra, and then we invent the microscope and find out that these polyhedra occur all over the natural world - in crystals, in the shape of viruses, and so forth - and no other ones do. That's a pretty good indication that $\mathbb{R}^3$ is a good model of at least some parts of the physical world. The way I think this result should be interpreted is that, even if you believe that there are only a finite number of possible locations in the physical universe (or whatever), the "mesh size" of the universe is small enough that we can take a limit and work in the infinitary setting, which boils down to discarding some negligible error terms.
Terence Tao's post on hard and soft analysis may also be of interest. I also want to quote somebody who, on MO, once said something like "do we ever talk about the infinite, or do we only talk about symbols that talk about the infinite?"
