On the problem of stability for higher-order derivative Lagrangian systems in Letters in Mathematical Physics (1987) may have the desired level of rigor (see Theorem 1).
The proof of the theorem is a straightforward few lines of algebra, so I would think that most derivations in the literature have the required level of rigor. All of this refers to the basic fact that higher-than-first-order time derivatives in the Lagrangian produce linear momentum terms in the Hamiltonian. The conclusion that this implies an instability goes beyond Ostrogradsky, is not part of the theorem, and is not true in general. (See arXiv:2403.19777 for counter examples.)
A major line of research is to find conditions on the Lagrangian that allow for higher order derivatives without compromising the stability, see Ghost-free theories with arbitrary higher-order time derivatives.