Mathematics embeds in natural language inheriting the grammatical structure of its 'host'. So mathematical constructions (eg. sets) are nouns, relations (like $\le$, $\in$ and $=$) are verbs, properties are adjectives and quantifications like $\forall x$ are adverbial phrases. The usual rules of grammar of the host natural language, including punctuation rules, now extend naturally.
So, for example, the sentence after this one is a perfectly good self-contained sentence so it should end with a period. $(x+1)^2>x^2+2x$.
