1. They assume the existence of objects that they don't construct.
2. Their logic is flawed.
3. They experiment (with particles and such) and assume that if it works enough times then it is true.
4. They assume that "reasonable" mathematical conjectures are true without bothering to be sure.
5. They don't have axiomatized definitions, and rely on vague notions.

To varying degrees, all five of these are the case. However, it's important to note that by and large physicists get the right answers. This is what makes physics (and all the other sciences, really) such a vital source of inspiration for mathematics -- results which are false on a straightforward reading of the scientists' language may actually hold for every "reasonable" or "nice" situation that a scientist is interested in. This means that there is often a lot of really cool mathematics lurking in the the formal characterization of what "nice" means.

A few examples:

• Eg, nonstandard analysis and synthetic differential geometry can be seen as ways of taking the language of infinitesmals that physicists pervasively use and putting it on a formal footing.
• People often say that a string like 0101001011000110111 is "more random" than one like 0000000000000000000. This statement doesn't make sense interpreted in the usual terms of probability theory, but does make sense when interpreted in terms of Kolmogorov complexity (and leads to ideas like the minimum description length principle of statistics).
• In computer science, people often say things like "merge sort and bubble sort are different functions". This doesn't make sense when you think of a function as a graph, but does make sense when you think of them as operations -- and taking that viewpoint seriously yields intuitionistic logic.