Others have articulated well why giving such a talk may be good for your career. I would like to mention a somewhat different motivation. It is frequently lamented that mathematics is over-specialized, that mathematicians cannot understand papers written in a field even slightly different from their own, etc. I believe that your experience at this conference illustrates one reason why this state of affairs exists: It's very comfortable to associate only with people in your own clique.
I find that intentionally venturing outside my comfort zone helps me develop a broader view. I sometimes discover that something that my sense of what is important and what is not has been unduly influenced by fashion rather than objective truth. Becoming acquainted with how people from a different clique think also improves my ability to explain and exposit, because it forces me to abandon the crutch of assuming that my audience already understands (and is interested in!) most of what I want to say.
If you are content with being a specialist who spends all of his or her time talking with other specialists, then avoiding conferences in other areas may be a reasonable thing to do. However, I think that there are many benefits to developing a broad grasp of many different areas.