No. "Theorem 2.4" is a title, hence capitalized. But the word "lemma" in "the previous lemma" is simply a non-proper noun, hence uncapitalized in English. In both your examples I would use the second version.
Although I wrote the above paragraph as though it were definitive, it is of course only my own opinion. There's (almost) no such thing as universally accepted usage.
Also, although I would write "The proof is postponed to Section 4," I'm not too bothered by "... to section 4." I have no good reason for being less rigid about this than about theorem capitalization. I've also been known to be inconsistent about things like "the Hahn-Banach theorem".
