As long as *connected* groups of isometries are concerned, Grassmann manifolds are symmetric spaces,
so the identity component of its isometry group
is $G$ in its symmetric presentation $G/H$ ($G$ connected) as a homogeneous space, namely, $SO(n)$ for $n$ odd and $SO(n)/\mathbf Z_2$ for $n$ even in the real case,
and $PU(n)=SU(n)/\mathbf Z_n$ in the complex case.
(Note that $U(n)$ acts on the left on the Grassmannian with a $U(1)$-kernel (its center), so the effectivized group is the projectivization
$PU(n)$. Moreover the center of $U(n)$ meets
$SU(n)\subset U(n)$ along its center, which consists of $\omega I$ where $\omega$ is an $n$-th root of unit.)

Further, Cartan described the full isometry groups of symmetric spaces, and an explicit result is easy to figure out in the case of Grassmann manifolds.
I do not remember now, but you can find Cartan's description in the book of O. Loos on symmetric spaces, the second volume. I tend to agree with Ryan when he writes that in the case of Grassmann manifolds, the full isometry group should be
$G\times N_G(H)$.

About Stiefel manifolds: with the metric you describe, they are normal homogeneous spaces
$G/H$, i. e. have the metric induced from a
bi-invariant Riemannian metric on $G$. There is a recent paper by S. Reggiani with a very effective way of computing the identity component of isometry groups of normal homogeneous spaces
in here.

**Added**: I looked up Loos, "Symmetric spaces, II", Theorem 4.4 and the ensuing Table 10 on page 156 for the full isometry group of the real and complex Grassmannians. If I understand correctly, indeed in the case of complex Grassmannians

$SU(n)/S(U(p)\times U(n-p))$, every isometry comes from left multiplication by elements from $SU(n)$ except for two cases: an isometry induced by complex conjugation; and mapping a $p$-plane to its orthogonal complement in case $n=2p\geq4$. In the case of
real unoriented Grassmannians $SO(n)/S(O(p)\times O(n-p))$, every isometry comes
from left multiplication by an element of $O(n)$ except for: mapping a $p$-plane to its orthogonal complement in case $n=2p\geq4$; the symmetric group $S_3$ in case $n=2p=8$,
coming from outer automorphisms of $\mathfrak{so}(8)$.