I tend to respond with something along the lines of, "Well, that's a shame. I hope you found something you were good at?" I've found it's easier to interface with non-mathematicians if you're willing and able to talk about whatever it is they do, rather than hope that they'll be interested in what you do...