It's hard to see this question as something specifically relevant to mathematics. That is, it sounds pretty much the same as asking `What are good ways of carrying on a conversation?' Of course there's no simple answer to that either. To the extent that I'm able to give anyone advice, I would simply call for general sympathy. If you've already established a reasonable dialogue, I don't see that the response you mention would have to carry any problematic connotations. For example, asking plenty of sincere questions about the other person's interests would be one easy way of eventually having a serious discussion about your own.
I would like to remark that the prevalence of a negative reaction to the mathematical profession appears to me greatly exaggerated. I can very honestly say I've never encountered it myself in any social situation. Unfavorable and unrealisic comparison to the arts appears with surprising frequency, as in Lockhart's Lament. But I wonder if such commentators have ever compared the income or employability of an average mathematician with that of an average artist, musician, or writer. My feeling is we are often misled by the fact that the stars in art, music, or literature are so much more prominent than their mathematical counterparts. I'd rather not do it now, but I believe it's rather easy to establish that mathematicians do very well on the average, and this is generally a good thing about the profession. Average status is alright, and we give our stars due respect without overinflating them. The absence of publically recognizable gloss keeps us humble enough to be suitably happy. One could say the same for academic apitutude in general, brought up in one of the answers above, which again is quite well-rewarded on the average. Muddled thinking surrounding 'creativity' has been detrimental to education in many ways, but it's really quite sad if the rhetoric keeps even my fellow mathematicians from recognizing their overall good fortune! (Incidentally, I won't go into them here, but I also have reasons to consider mathematicians especially fortunate even among academics.)
By the way, I really don't like the answers that suggest blaming former teachers. This is an unfortunately common and irresponsible response to a complex question. I don't want to express this in too harsh a manner, but I might ask Qiaochu if he'd heard the teachers' side of the story before reaching his conclusions. Most primary and secondary school teachers I know of are dedicated people working under frequently difficult circumstances, doing what they can to teach basic and necessary skills. Perhaps their own skills are not optimal, but neither are mine.
Maybe I should nevertheless conclude this ramble with a brief answer to the original question: With genuine sympathy, say `Oh, I'm sorry to hear that,' and proceed with good humor.
P.S. I apologize for the moralistic tone of this reply. My impression is many users of this site are quite young, and I'm still a Confucian at heart.
Added, 18 November:
I feel compelled to add a few comments given the seriousness of the topic. That is, I would like people to consider the possibility that many perceived problems arise simply because of the overwhelming importance of mathematics.
Jonah Sinick made a number of thoughtful comments in a separate email, and one point he brought up was that the math he learned prior to the university looks nothing like the kind he knows and loves now. This is probably true. But since the comparison with music or sports is made frequently, consider how much a beginner at the violin practicing the scales resembles a professional performing Beethoven. And then how about running around the field for hours to build up stamina vs. the antics of NBA players? Obviously, there are many things whose mastery requires patient tolerance of the basic skills. Why then so much more perceived difficulty with regard to mathematics? This is because the entire population must learn it to a certain level, much higher, for example, than typical violin skills. Society in general at present considers mathematics that important, and much of the dysfunction flows out of this situation. I leave it to you to imagine a scenario where every child had to perform basic violin pieces as well as basic arithmetic operations, regardless of their inclination or talent. And then imagine we needed as many violin teachers as teachers of mathematics.
I hope none of you wish to argue then that mathematics is in fact not so important as to require such massive social investment. This is not because it runs against our interest, but because that's a really difficult case to make in our modern age, and probably too long to fit into the margins of this webpage. In short, don't blame teachers, don't blame the children, don't blame mathematicians. Blame industrial civilization, if you're so inclined.
The main thing I find surprising in many such discussions is that everyone realizes playing the violin or basketball well requires many hours of boring practice that lead up to the joy that comes with the different levels of mastery. But mathematics, and perhaps some other academic subjects, are supposed to be constantly concerned with being 'fun'. And then, I don't see the violinists themselves viewing the lack of inspiration in scales as a fundamental problem.
Of course I hope as many teachers as possible are able to convey some real sense of joy in regard to the mathematics they teach, and various notions of creativity can be occasionally useful and inspiring. But given the sheer scale of the task at hand, I wouldn't hope for methodological ingenuity to bring about overall improvement at any easily measurable pace.
Yet another addition.
I realize as I reread my own paragraphs that I've been explaining mostly one side of the story. So I'm adding a link to a short article that attempts to present a more balanced perspective.