It seems to me that frequently people will say this because they find mathematics intimidating. I don't think they're trying to be rude.
I try to get the idea across that mathematics is not always as complicated as it seems. For instance, the average adult may not know the term "integration", but a child can easily count the number of squares underneath a curve.
