Here is a direct proof along the lines of the standard proof of the Cayley–Hamilton theorem.
The following lemma combining Abel's summation and Bezout's polynomial remainder theorem is immediate.
Lemma Let $A(\lambda)$ and $B(\lambda)$ be matrix polynomials over a (noncommutative) ring $S.$ Then $A(\lambda)B(\lambda)-A(0)B(0)=\lambda q(\lambda)$ for a polynomial $q(\lambda)\in S[\lambda]$ that can be expressed as
$$q(\lambda)=A(\lambda)\frac{B(\lambda)-B(0)}{\lambda}+\frac{A(\lambda)-A(0)}{\lambda}B(0)=A(\lambda)b(\lambda)+a(\lambda)B(0), \quad a(\lambda),b(\lambda)\in S[\lambda] \qquad (*)$$
Let $A(\lambda)=A-\lambda I_n$ and $B(\lambda)=\operatorname{adj} A(\lambda),$ then $A(\lambda)B(\lambda)=\det A(\lambda)=p_A(\lambda)=p_0+p_1\lambda+\ldots p_n\lambda^n$ is the characteristic polynomial of $A,\ A(0)B(0)=p_0,$ and $q(\lambda)=p_1+\ldots+p_n\lambda^{n-1}.$ Applying $(*),$ we get
$$q(\lambda)=(A-\lambda I)b(\lambda)-\operatorname{adj} A \qquad (**) $$ for some polynomial $b(\lambda).$ Specializing $\lambda$ to $A,$ we conclude that
$$q(A)=-\operatorname{adj} A\qquad \square$$