Is it worthwhile to give off-topic talks? I am a graduate student. Occasionally for some reason I am asked to give a talk on my research at a conference whose stated purpose is almost completely unrelated to my research. To preserve my anonymity, I won't say what I actually do, but as an example: Imagine I do research in some particular aspect of exterior differential systems, and I am asked to speak at a special session on some particular aspect of numerical analysis. 
In the past when I have done this, it has not been a pleasant experience. I sit through a bunch of talks I don't understand, try to socialize with a group of people who already know each other and who I'll likely never see again, and then give my talk, which I feel is seen as irrelevant. 
But I am still encouraged (by my advisor and others) to give such talks, because it is good for my career. In particular, I've been told that it's good to advertise my work to a broad audience, and that people in my audience may someday referee my papers. This seems very unlikely to me, but I am uncertain.
Is giving such a talk really a worthwhile endeavor? 
 A: Rather than an unqualified "go anyway" or "stay home", I would suggest exploring why you are getting invited to these conferences.  Unless the conference is a total sham (such things do exist; does it have a hefty registration fee?), the organizers are noted experts in (following your example) numerical analysis, and they have some reason to think your talk would be interesting to the audience.  If it isn't clear to you why that is, you should try to find out.
Perhaps there are connections between numerical analysis and exterior differential systems that you're not aware of.  Talk to your advisor, and also try to find a numerical analyst to talk to (your advisor may know someone if you don't).  Ask them, "What aspects of my work would be most interesting to numerical analysts?"  If you can learn about what these connections are, you can use them as a guide for what you should highlight in your talk.
You could also ask the organizers who invited you.  "Thanks for the invitation.  I've recently been working on topic X, do you think that would fit well with the theme of your session?"  They might offer a suggestion as to aspects that might interest their intended audience (especially the other invitees), and you can plan your talk accordingly.  Or they can say "We were actually hoping you'd speak on your recent work on Y", which gives you another hint.
Of course, there's the remote possibility that you've been invited by mistake.  Do you have a similar name to a famous numerical analyst?  If this should be the case, contacting the organizers as above will help them figure it out and gently fix the error without embarrassing anyone.  (If they say "We were hoping you would speak on Y", and Y is something you've never heard of, then this would suggest a mistake has been made.)
Like the other answers, I don't think you should stay away just because you're not 100% comfortable; one can often get a lot out of conferences that seemed unlikely.  But if you really have absolutely no idea why you're there, and therefore your talk is completely irrelevant, I agree with you that it doesn't seem like it will give you positive exposure.  So my answer is neither "accept regardless" or "stay away", but "find out why you're there" and "give a more relevant talk"!
A: Others have articulated well why giving such a talk may be good for your career.  I would like to mention a somewhat different motivation.  It is frequently lamented that mathematics is over-specialized, that mathematicians cannot understand papers written in a field even slightly different from their own, etc.  I believe that your experience at this conference illustrates one reason why this state of affairs exists: It's very comfortable to associate only with people in your own clique.
I find that intentionally venturing outside my comfort zone helps me develop a broader view.  I sometimes discover that my sense of what is important and what is not has been unduly influenced by fashion rather than objective truth.  Becoming acquainted with how people from a different clique think also improves my ability to explain and exposit, because it forces me to abandon the crutch of assuming that my audience already understands (and is interested in!) most of what I want to say.
If you are content with being a specialist who spends all of his or her time talking with other specialists, then avoiding conferences in other areas may be a reasonable thing to do.  However, I think that there are many benefits to developing a broad grasp of many different areas.
A: I support a nice Joël’s  answer.  
I think that success depends on communication and social activity, but personally I prefer to do more explicit carrier steps than those giving you only a hope that “maybe somebody somewhere will remember me”. I consider that if after a few months after the conference your listener will remember from where you are and that you are dealing with exterior differential systems, it will be very good. We may check this if we ask ourselves what we remember about conference speakers which were far from our mathematical interests, and I think that most of the mathematicians are narrow specialists. 
I think that Internet is a good way to general (and, in particular to working and carrier) acquaintances and communications. In particular, we can use LinkedIn or MathOverflow for this purposes. Personally I gathered a material for two joint articles after few months at Mathematics Stack Exchange. When I was a graduate student and my English was more weak, I even had a sample letter to ask other mathematicians for papers. And I solved some open problems.
I think that success depends on many factors. For instance, I rarely visited conferences outside my city but I have 17 coauthors and big mathematical correspondence. Also my friend told me that I must attend conferences and directly contact with people to be invited, but I was invited to China when I had reviewed an article written by a Chinese professor. 

Concerning the mentioned differences between American and Russian scientific styles of work, I think that the style is strongly depending on a personal choice. 
Andrew Wiles had rejected his partisipation in conferences and was proving Fermat’s Last Theorem on his own (see for the details Chapter 6 of Simon Singh’s “Fermat's Last Theorem”, which fragments are here and here).
Paul Erdős “spent most of his life as a vagabond, traveling between scientific conferences and the homes of colleagues all over the world. He would typically show up at a colleague's doorstep and announce “my brain is open”, staying long enough to collaborate on a few papers before moving on a few days later. In many cases, he would ask the current collaborator about whom to visit next”. He was generoulsy sharing his mathematicas ideas with community and was easily responding to others’s ideas. He died from a heart attack attending a conference in Warsaw and he had in his pocket a ticked to Vilnius, where should be his next conference.
Personally I was bred in L’viv topological school which is a branch of a topological school from Moscow State University. I was very disliking conferences and preferring not to leave my city. For instance, when I was in secondary school, I even skipped a participation in Soviet Union mathematical olympiad, because I was not interested in it, althouh I had a gold medal in Ukraine. :-) But now, living in a real world, which seems to be similar and to have similar problems both in America and Russia, I am forced to do scientific carrier in order to make a possibility to be a scientist. :-(
As a resume I recommend you to choose (or form your own) a scientific style that works better for you, gives you growth and is good for you.
A: Yes.  You'd crazy not to accept these invitations if they don't create a serious difficulty  for you in some other realm (family/teaching/etc.).   Once you have a TT position, you can afford to be a bit choosy, but if you're a grad student and you're getting recurring requests to go to conferences which are outside your area, you should be praising [insert your deity of choice here].  Awkwardness at conferences is just par for the course as a grad student; you're only going to get to know some critical mass of people by going to conferences with lots of people you don't know.
I think the other way to look at this is that there is a not huge, but also not tiny chance, that you will learn something totally transformative for your work at one of these conferences, or make an crucial personal connection for the next stage of your career, or someone will see your talk and have a invaluable comment to make on another way of seeing it.  Even if the chance of these things is, say, 20%, well, that's better odds than the lottery.  
From a practical perspective, even if your talks aren't thrilling to people, having your name in people's brains will be key for the next stage of your career.  The number of applications for postdoc positions is totally overwhelming for many purposes, so having someone at the school already know your name is a good start.
A: I'm guessing you're unlikely to get a 'no' answer, so let me quote here a paragraph of V.I. Arnold:

A few years ago, I was participating in an International Science Foundation (ISF) meeting in Washington, DC. This organization distributes grants to Russian scientists. One American participant suggested support for some Russian mathematician because “he is working in a good American style.” I was puzzled and asked for an explanation. “Well,” the American answered, “it means that he is traveling a lot to present all his latest results at all our conferences and is personally known to all experts in the field.” My opinion is that ISF should better support those who are working in the good Russian style, which is to sit at home working hard to prove fundamental theorems which will remain the cornerstones of mathematics forever!

An Interview with Vladimir Arnol'd, Notices of the AMS, Vol 44 No 4, p.436
A: Perhaps I should clarify this "answer" in light of the downvote as well as the OPs comment. 
I don't intend to cause offense or make light of your situation. I went through a similar enough phase as a graduate student, and with high probability, so did everyone else who is saying "of course you should go and give talks wherever people let you". So let me point out three concrete and (I hope) helpful things for the next time you are in this situation.


*

*One of the signs that you have "made it" in the math business is that you start getting invited to give colloquium talks. These are far scarier to plan than even numerical analysis talks, because there is almost nothing you can say that the whole room will "get"  even though it may be filled to the brim with experts. This is an inevitable outcome of specialization -- one person's basic facts are another person's alien magic. The sooner one gets used to adapting to such situations (both as a speaker and as a listener), the smoother this transition becomes.

*Typically, you have access to what people are going to say before they say it. At the end of the day, we are all responsible for our own ignorance -- read the introductions to the appropriate papers/preprints of the people who are giving talks in your alien conferences whenever possible. The less blind you walk in, the less blind you'll walk out.

*And finally, if all has gone to hell and you are completely lost, you can at least focus on the way the speaker is conveying information rather than the content. Watching other people give good/bad talks is a spectacular way of finding your own voice and learning what to do/avoid in your own lectures.
Good luck, and sorry about my tongue-in-cheek knee-jerk response from earlier. I think you will find that these things have a way of working out.

I am an undergraduate student. Occasionally for some reason I am asked to take a class at my institution whose stated purpose is almost completely unrelated to my major. To preserve my anonymity, I won't say what I actually do, but as an example: Imagine I am majoring in computational biophysics, and I am asked to take a course in linear algebra.
In the past when I have done this, it has not been a pleasant experience. I sit through a bunch of lectures I don't understand, try to socialize with a group of people who already know each other and who I'll likely never see again, and then take my finals, which I feel is seen as irrelevant.
But I am still encouraged (by my advisor and others) to take such classes, because it is good for my career. In particular, I've been told that it's good to learn diverse techniques and ideas, and that I may someday need them in my career. This seems very unlikely to me, but I am uncertain.
Is taking such classes really a worthwhile endeavor?
A: My answer is a clear no. If your experience at a type of conference if that you hate being there, then don't go again, at least for a few years -- then you can reconsider. Save your time, energy, travel money, any of your scarce resource for more interesting purpose, in particular for working on your PhD.
In general, if anything that you're not strictly obliged to do in the mathematical world really bores you, then don't do it. Try to protect your love for mathematics.
In the long run, I am sure that such an attitude will help your career.
Note that I am not saying that you shouldn't go to conferences in other fields
because they are in other fields, but just because that, according to your post, at this time of your life and career, you simply don't like going there.     
