The tricky thing about that response is that there's often some pride in not being good at math, perhaps because they've gotten so far with out having to know any math (so they must be pretty good, right?). More often than not, what the person is getting at is that they were right when they said to themselves in grade school "I'm never going to need this."
I size the person up, and if I think the answer will be an emphatic "YES", I respond with something like, "but you can read, right?" Then I try to draw a parallel between math and reading. Neither of them (in much quantity) is strictly necessary for getting by, or even to be successful, but if you read (or think mathematically) often, it enriches your life. At this point, their curiosity is usually piqued, so I'll try to steer the discussion to how your understanding of the world might be improved by thinking about probability, symmetry, or abstraction (depending on what I think will best engage the person). I try to get across that doing mathematics can be for pure enjoyment or curiosity with examples of interesting or counterintuitive elementary results.