Many (most? all?) North American graduate programs have some form of qualifying exam (which goes by different names at different institutions) whose goal is to establish a baseline knowledge of the kind that you ask about. Typical topics are algebra and real and complex analysis.
However, there is no law (natural or human) saying that a person will remember what they have once been taught, and if one's goal (in a colloquium, or equivalently, any lecture to non-sepcialists) is to be understood, it will not be any use to appeal to some alleged common knowledge specified in a document, or a qualifying exam syllabus, or anywhere else.
I think a more realistic solution is to develop mechanisms for explaining ideas which appeal to a wide range of audience members, and include many different ways for them to try and understand what you are talking about. This is never easy, unforutnately, but it can become easier with practice. I think one thing to remember is that simple geometric ideas are often easier to communicate than algebraic ones, and are (in my experience) more likely to appeal to a wide range of audience members. Also, I think one should be especially receptive to the idea that audience members will be confused by your notation, and thus one should adopt simple notation and be prepared to explain it. (So my sympathies are with you in your attempt to divine the meaning of $\hat{f}(t)$!)
[Added in response to edit to question:] The best way to improve the average level of talks is, I think, to train graduate students to give good talks. In general, the mathematical community is sensitive to the quality of talks: e.g. job candidates can fail an interview because of a badly delivered job talk; an important criterion for plenary speakers at AMS events is that they can give good talks; colloquium speakers are typically selected in part on the basis of their ability to give good talks. So the ability to give good general audience talks is an important professional skill, and we should do our best to teach it (along with all the other important professional skills that we teach to our graduate students).