This is something I've never paid attention to until graduate school, but virtually every book uses the convention that formulae in display mode are part of the text. Every Springer text for instance uses these conventions.
If we define the function $f:\mathbb{R}\rightarrow\mathbb{R}$ by
$$ f(x) = e^x, $$
then we can place a comma after the definition to indicate a pause one might take if speaking such a sentence. We could also have defined the function by
$$ f(x) = \sin(x). $$
As this last definition was the end of a sentence, it ought to have a period. Finally we could also have
$$ |f(x) - f(x_0)| < \varepsilon $$
whenever $|x - x_0| < \delta$. Here, no punctuation was needed.
There are exceptions: Spanier's Algebraic Topology doesn't follow these conventions, but Hardy does, and all modern books that I've read do. Unless I pay attention, I don't even notice the punctuation.
