Following up on (1) in Ben McKay's answer, taking $\operatorname{Rm}$ to be a
covariant $4$-tensor, $\nabla^{k}\operatorname{Rm}$ is a (covariant)
$(k+4)$-tensor, i.e., a section of $\bigotimes^{k+4}T^{\ast}M$. So in
coordinates, $|\nabla^{k}\operatorname{Rm}|^{2}=g^{i_{1}j_{1}}\cdots
g^{i_{k+4}j_{k+4}}\nabla_{i_{5}\cdots i_{k+4}}^{k}R_{i_{1}i_{2}i_{3}i_{4}
}\nabla_{j_{5}\cdots j_{k+4}}^{k}R_{j_{1}j_{2}j_{3}j_{4}}$ or with respect to
an orthonormal frame $\{e_{i}\}$,
$$
|\nabla^{k}\operatorname{Rm}|^{2}=(\nabla_{e_{i_{5}}\cdots e_{i_{k+4}}}
^{k}\operatorname{Rm})(e_{i_{1}},e_{i_{2}},e_{i_{3}},e_{i_{4}})(\nabla
_{e_{i_{5}}\cdots e_{i_{k+4}}}^{k}\operatorname{Rm})(e_{i_{1}},e_{i_{2}
},e_{i_{3}},e_{i_{4}}),
$$
summing over repeated indices.
For a $C^{\infty}$ metric $g$ on a closed manifold $M$, we have $C_{m}
\doteqdot\sup_{M}\left\vert \nabla^{m}\operatorname{Rm}\right\vert <\infty$;
but these constants also depend on $g$. Given $K$, under the restriction
$\left\vert \operatorname{Rm}\right\vert \leq K$, it is easy to construct
metrics with $\left\vert \nabla\operatorname{Rm}\right\vert $ arbitrarily
large. For example, on the noncompact $\mathbb{R}\times S^{1}$ consider
$g_{\varepsilon}\doteqdot dr^{2}+\left( 1+\varepsilon^{2}\sin\left(
r/\varepsilon\right) \right) ^{2}d\theta^{2}$, where $\varepsilon\in
(0,1/2]$. Then $R\left( g_{\varepsilon}\right) =\frac{2\sin\left(
r/\varepsilon\right) }{1+\varepsilon^{2}\sin\left( r/\varepsilon\right) }$
satisfies $\left\vert R\left( g_{\varepsilon}\right) \right\vert \leq
\frac{8}{3}$. Note that $\left\vert \nabla R\left( g_{\varepsilon}\right)
\right\vert =|\frac{\partial}{\partial r}R\left( g_{\varepsilon}\right)
|=\frac{2}{\varepsilon}\frac{\left\vert \cos\left( r/\varepsilon\right)
\right\vert }{\left( 1+\varepsilon^{2}\sin\left( r/\varepsilon\right)
\right) ^{2}}$, so that $\sup_{\mathbb{R}\times S^{1}}\left\vert \nabla
R\left( g_{\varepsilon}\right) \right\vert \geq\frac{2}{\varepsilon}$. For
$\varepsilon=j^{-1}$ we can take quotients to yield a sequence of metrics on
the compact $S^{1}\times S^{1}$ with $|R(g_{j^{-1}})|\leq\frac{8}{3}$ and
$\sup_{S^{1}\times S^{1}}\left\vert \nabla R\left( g_{j^{-1}}\right)
\right\vert \geq2j$.
The idea behind the derivative of curvature estimate is ubiquitous in
geometric analysis and goes back to Bernstein in PDE and Bochner in geometry.
Given a tensor $T$, $\frac{1}{2}\Delta\left\vert T\right\vert ^{2}=\left\vert
\nabla T\right\vert ^{2}+\left\langle \Delta T,T\right\rangle $; if $T=\nabla
U$ then we use $\Delta\nabla U=\nabla\Delta U+\left[ \Delta,\nabla\right]
U$, where $\left[ \Delta,\nabla\right] $ involves curvature. For example,
the fundamental lemma of geometric analysis is $$\frac{1}{2}\Delta\left\vert
\nabla u\right\vert ^{2}=\left\vert \nabla^{2}u\right\vert ^{2}+\left\langle
\nabla\left( \Delta u\right) ,\nabla u\right\rangle +\operatorname{Ric}
\left( \nabla u,\nabla u\right) . $$
Under Ricci flow for $g$, modulo
$\operatorname{Ric}\ast T^{\ast2}$ terms, we have $\frac{1}{2}(\frac{\partial
}{\partial t}-\Delta)\left\vert T\right\vert ^{2}=-\left\vert \nabla
T\right\vert ^{2}+\langle(\frac{\partial}{\partial t}-\Delta)T,T\rangle$.
To exhibit the idea, consider a solution to the heat equation $\frac{\partial
f}{\partial t}=\Delta f$ with $\left\vert f\right\vert \leq K$ on a static manifold
with $\operatorname{Ric}\geq0$. We have $\frac{1}{2}(\frac{\partial}{\partial
t}-\Delta) ( f^{2}) =-\left\vert \nabla f\right\vert ^{2}$ and
$\frac{1}{2}(\frac{\partial}{\partial t}-\Delta)\left\vert \nabla f\right\vert
^{2}\leq-\left\vert \nabla^{2}f\right\vert ^{2}$. Assuming the maximum
principle holds, $\frac{1}{2}(\frac{\partial}{\partial t}-\Delta)(t\left\vert
\nabla f\right\vert ^{2}+\frac{1}{2}f^{2})=-t\left\vert \nabla\nabla
f\right\vert ^{2}\leq0$. Hence $\left\vert \nabla f\right\vert ^{2}\leq
\frac{K^{2}}{2t}$ for $t>0$.
For Ricci flow, the computations are similar: $\frac{1}{2}(\frac{\partial
}{\partial t}-\Delta)\left\vert \operatorname*{Rm}\right\vert ^{2}
\leq-\left\vert \nabla\operatorname*{Rm}\right\vert ^{2}+C_{0}\left\vert
\operatorname{Rm}\right\vert ^{3}$ and $\frac{1}{2}(\frac{\partial}{\partial
t}-\Delta)\left\vert \nabla\operatorname*{Rm}\right\vert ^{2}\leq-\left\vert
\nabla^{2}\operatorname*{Rm}\right\vert ^{2}+C_{1}\left\vert \operatorname{Rm}
\right\vert \left\vert \nabla\operatorname*{Rm}\right\vert ^{2}$, so that
$F=t\left\vert \nabla\operatorname*{Rm}\right\vert ^{2}+\left\vert
\operatorname*{Rm}\right\vert ^{2}$ satisfies
$$
\frac{1}{2}(\frac{\partial}{\partial t}-\Delta)F\leq(C_{1}t\left\vert
\operatorname*{Rm}\right\vert -\frac{1}{2})\left\vert \nabla\operatorname*{Rm}
\right\vert ^{2}+C_{0}\left\vert \operatorname*{Rm}\right\vert ^{3}.
$$
Assume $M$ is closed and $\left\vert \operatorname*{Rm}\right\vert \leq K$. Applying the maximum
principle yields $\left\vert \nabla\operatorname*{Rm}\right\vert ^{2}\leq
\frac{C_{2}K^{2}}{t}$ for $t\in(0,(2C_{1}K)^{-1}]$.
Following up on Deane Yang's answer, Shi's first derivative estimate is local and says that if
$g\left( x,t\right) $, defined (only locally) in $B_{g\left( 0\right) }(p,r)\times
\lbrack0,T]$, satisfies $\left\vert \operatorname*{Rm}\right\vert \leq K$,
then $\left\vert \nabla\operatorname*{Rm}\right\vert \leq C_{n}K\left(
\frac{1}{r^{2}}+\frac{1}{t}+K\right) ^{1/2}$ in $B_{g\left( 0\right)
}(p,\frac{r}{2})\times(0,T]$, where $C_n$ depends only on $n$. E.g., taking $r=cK^{-1/2}$ and $t=cK^{-1}$, we
obtain $\left\vert \nabla\operatorname*{Rm}\right\vert (x,cK^{-1})\leq
C_{n}K^{3/2}$ in $B_{g\left( 0\right) }(p,\frac{c}{2}K^{-1/2})$.
One way to prove Shi's estimate (localizing in a way
following Hamilton's Formation of Singularities paper) is as follows. Let
$G=t(16K^{2}+\left\vert \operatorname*{Rm}\right\vert ^{2})\left\vert
\nabla\operatorname*{Rm}\right\vert ^{2}$. One computes that $(\frac{\partial
}{\partial t}-\Delta)G\leq\frac{1}{t}\left( -c_{3}K^{-4}G^{2}+C_{4}
K^{4}\right) $ for $t\in(0,K^{-1}]$. Essentially, because of the good
quadratic term involving $G^{2}$ on the right side, this equation is amenable
to localization, i.e., multiplication by a cutoff function.
See Bing-Long Chen's paper using Perelman's time-dependent localization to prove that any
complete ancient solution to the Ricci flow must have $R\geq0$ ($R>0$ unless
$\operatorname{Ric}=0$). In dimension 3, by a localization inspired by the Hamilton-Ivey estimate, Chen proved that any complete ancient solution must have nonnegative sectional curvature.