No for $n=3,4$. Indeed, consider the following property $L(n)$, for a prime $p$: there are integers $k, m_1,\dots,m_n$ with $k,\gcd(m_1,\dots,m_n)$ coprime to $p$ such that $\sum_{i=1}^n m_i^2=k^2p^2$.

For $n\ge 2$ I claim that $\mathrm{SO}(n,\mathbf{Q})$ is contained in $\mathrm{SO}(n,\mathbf{Z}_p)$ if and only $L(n)$ fails.

Indeed, suppose that the inclusion fails. Then there is a column in some such matrix $\mathrm{SO}(n,\mathbf{Q})$ that is not in $\mathbf{Z}_p$. That is, it writes as $(m_1/k,\dots,m_n/pk)$ with $p$ not dividing some $m_i$. Then $\sum m_i^2=k^2p^2$ and $L(n)$ holds. Conversely, suppose that $L(n)$ holds. So there are $k,m_i$ as given. Then by Witt's theorem, there is a matrix in $\mathrm{O}(n,\mathbf{Q})$ whose first column is $(m_1/kp,\dots,m_n/kp)$. Changing the sign of the second column, we can suppose it has determinant one. So the inclusion fails.

Hence, if $L(n)$ holds and $L(n-1)$ fails, then $\mathrm{SO}(n,\mathbf{Q})$ is not generated by stabilizers of basis elements.

It remains to see that 

 - for $p=3$, $L(3)$ holds and $L(2)$ fails
 - for $p=2$, $L(4)$ holds and $L(3)$ fails.

Indeed for $p=3$, $L(2)$ fails because $0$ is not a sum of two nonzero squares modulo 3 (this works for any odd prime $p$ in $4\mathbf{Z}+3$). Also $L(3)$ holds: $1^2+2^2+2^2=3^2$. Concretely, any matrix in $\mathrm{O}(3,\mathbf{Q})$ whose first column is $(1/3,2/3,2/3)$ is not a product of matrices fixing basis elements.

For $p=2$, $L(3)$ fails because $0$ is not a sum of three not-all-even squares modulo 4. And $L(4)$ holds because $1+1+1+1=2^2$.