Probably the simplest such manifold is $S^1 \times S^2$, whose diffeomorphism group has the homotopy type of $O(2) \times O(3) \times \Omega SO(3)$. This has $\pi_2$ equal to $\pi_2\Omega SO(3)=\pi_3 SO(3) = {\mathbb Z}$. The $\Omega SO(3)$ term is realized by rotating the $S^2$ slices of $S^1\times S^2$ by an element of $SO(3)$ that varies as one goes around the $S^1$ factor of $S^1 \times S^2$. This calculation was originally done in a paper of mine in the 1981 AMS Proceedings. An updated version of this paper is posted on my webpage.
Added later:
In higher dimensions, spheres provide interesting examples. By an elementary argument there is a homotopy equivalence $Diff(S^n) \simeq O(n+1) \times Diff_\partial(D^n)$ for all $n$, where the subscript $\partial$ denotes diffeomorphisms that restrict to the identity on the boundary of $D^n$. The question is whether $Diff_\partial(D^n)$ is contractible. The status of this is: true for $n\le3$, unknown for $n=4$, and false for each $n\ge 5$. The noncontractibility can be deduced from the sequence $$\cdots\ \to \pi_2 Diff_\partial(D^{n-2})\to \pi_1 Diff_\partial(D^{n-1})\to \pi_0Diff_\partial(D^n) = \Theta_{n+1}$$ where $\Theta_{n+1}$ is the group of exotic (n+1)-spheres and the equality $\pi_0Diff_\partial(D^n) = \Theta_{n+1}$ is assuming $n\ge 5$. Usually $\pi_0Diff_\partial(D^n)$ is nonzero since exotic spheres exist in most dimensions greater than $6$, and in the rare dimensions in which they don't exist one can appeal to known results about how far some elements of $\Theta_{n+1}$ pull back in the sequence above. For example, Cerf's pseudoisotopy theorem says the map from $\pi_1 Diff_\partial(D^{n-1})$ to $ \pi_0Diff_\partial(D^n)$ is surjective for all $n\ge 6$, so in particular $\pi_1 Diff_\partial(D^5)$ is nonzero since it maps onto $\Theta_7={\mathbb Z}/28$. A paper of Crowley and Schick posted on the arXiv last month shows there is a nontrivial element of $\Theta_{8k+2}$ that pulls all the way back to $\pi_{8k-6}Diff_\partial (D^7)$, hence $Diff_\partial(D^n)$ has infinitely many nontrivial homotopy groups for all $n\ge 7$.
Since the groups $\Theta_{n+1}$ are finite (apart perhaps from the unknown $\Theta_4$), these constructions don't give nontrivial rational homotopy groups of $Diff_\partial(D^n)$, but there is another construction that does, coming from algebraic K-theory. Using Waldhausen's big machine, Farrell and Hsiang in 1978 computed $\pi_iDiff_\partial (D^n) \otimes{\mathbb Q}$ in a stable range $n>>i$ to be ${\mathbb Q}$ for $i\equiv 3$ mod $4$ and $n$ odd, and $0$ otherwise. (This is the result mentioned in Vitali Kapovitch's answer.)