I find a far more common response to "I'm a mathematician" is "So you deal with really big numbers, right?"
MLevi's answer suggested a lot of people believe mathematics ends at calculus. In my experience more people don't realise mathematics extends beyond some notion of complicated sequences of arithmetic.
To a few people who replied with this or similar I asked them what they really thought I did with my day:
"I suppose you wake up and do some sums, maybe?"
