Your minimization problem is equivalent to \begin{equation*} \min_{R^TR=I}\quad\prod_{i=1}^p r_i^T\Sigma r_i, \end{equation*} and it can be shown (using Hadamard's determinant inequality and some more argumentation) that this minimum overall $p$ orthonormal tuples is achieved by choosing the $r_i$ corresponding to the smallest $p$ eigenvectors.
EDIT Here the details that complete my answer (because it seems that the OP already knew the answer and still posted this question...)
We know that for a nonnegative definite matrix $\Sigma$ with eigenvalues $\lambda_1,\ldots,\lambda_n$, and any orthonormal $R$,
\begin{equation*} \prod_{i=n-p+1}^n \lambda_i \le \det(R^T\Sigma R) \le \prod_{i=1}^p (R^T\Sigma R)_{ii} = \prod_{i=1}^p r_i^T\Sigma r_i. \end{equation*} We are trying to minimize the upper, bound and by choosing $R$ to be the matrix of the smallest $p$ eigenvectors, we can actually turn the first inequality into an equality. This proves the claim. The first inequality is a classic result on eigenvalues, while the second inequality is Hadamard's determinant theorem.
