I think the proof that the normal PDF is a PDF (i.e. integrates to 1) is beautiful and deserves to be in a probability course. Obviously, it's mostly calculus, so perhaps not elementary enough for what you have in mind, but it's really beautiful.