Let $X = (S^2\times S^2)/\mathbb{Z}_2$ where the $\mathbb{Z}_2$ action is generated by $(x, y) \mapsto (-x, -y)$. Note that $H_2(X; \mathbb{Z}) \cong \mathbb{Z}_2$, so every diffeomorphism acts trivially.

Consider the diffeomorphism $f : S^2\times S^2 \to S^2\times S^2$ given by $(x, y) \mapsto (x, -y)$. This descends to a diffeomorphism $g : (S^2\times S^2)/\mathbb{Z}_2 \to (S^2\times S^2)/\mathbb{Z}_2$. Letting $\pi : S^2\times S^2 \to (S^2\times S^2)/\mathbb{Z}_2$ be the universal covering map, we have a commutative diagram

$$\require{AMScd}
\begin{CD}
S^2\times S^2 @>{f}>> S^2\times S^2\\
@V{\pi}VV @VV{\pi}V \\
(S^2\times S^2)/\mathbb{Z}_2 @>{g}>> (S^2\times S^2)/\mathbb{Z}_2
\end{CD}$$

Taking $\pi_2$ of this diagram, we get a commutative diagram of abelian groups

$$\require{AMScd}
\begin{CD}
\mathbb{Z}\oplus\mathbb{Z} @>{f_*}>> \mathbb{Z}\oplus\mathbb{Z}\\
@V{\pi_*}VV @VV{\pi_*}V \\
\mathbb{Z}\oplus\mathbb{Z} @>{g_*}>> \mathbb{Z}\oplus\mathbb{Z}
\end{CD}$$

Note that $\pi_* = \operatorname{id}$ as $\pi$ is a covering map, but $f_*$ is given by $(a, b) \mapsto (a, -b)$. By commutativity, the same is true of $g_*$. In particular, $g_* \neq \operatorname{id}$ and therefore $g$ is not homotopic to the identity map.

Alternatively, note that $(S^2\times S^2)/\mathbb{Z}$ is orientable. As $\pi\circ f = g\circ\pi$, the maps $f$ and $g$ have the same degree, and it is easy to see that $f$ has degree $-1$. Again we see that $g$ is not homotopic to the identity map.

Geometrically, $X = \operatorname{Gr}(2, 4)$, the Grassmannian of unoriented two-planes in $\mathbb{R}^4$, and $S^2\times S^2 = \operatorname{Gr}^+(2, 4)$ the corresponding oriented Grassmannian. The diffeomorphisms $f$, $g$ are the maps given by $P \mapsto P^{\perp}$.