Having been to Japan a few times to work with my collaborators, it might well be that the hosts in your story just gave up to be polite, and willwould have a few laughs at the silly Westerners later in the drinking session that evening. (-:
I had a similar story. A Russian gentleman I met at a conference told me exactly the same thing Weil supposedly said. I remember him vividly because he would rise up after each talk and said something like: "Grothendieck would have done this, have you considered?". Remarkably, he asked no question after my talk, so we struck a nice conversation and went to the beach to swim (it was winter time in Korea). He was in better shape then I was despite being about 70 (apparently people from Russia have more experience swimming in the cold).
About your actual question, obviously being fit is important in all walks of life. Keep in mind that the physical activities one enjoys depend on many factors: countries of origin, class, race, social circles, etc. Mathematicians I know tend to like walking, hiking, biking. More seem to play soccer than tennis.
Talking about Serre, here is a video of him climbing a rock a few years back, from Robin Hartshorne's FB page. https://www.facebook.com/robin.hartshorne/videos/10200853358373274