I think the answer for the $Sl(n,Z)$ is positive if $n>1$. Let $D$ be a diagonal matrix, we ask if there's a $M$ such that $M-I$ is equivalent to $D$.($A$ is equivalent to $B$ if there are invertible $P,Q $such that $PAQ=B$.) This is same as asking if there's an $R$ in $SL(n,Z)$ s.t. $D+R$ is in $Sl(n,Z)$.(Reasoning:$ P(M-I)Q=PMQ-PQ$, and $PMQ$ is in $Sl(n,Z)$, $PQ$ is in $SL(n,Z)$, (neglecting the $\pm1$, which is not important)).
For any $P,Q$ in $Sl(n,Z)$, $D+R$ in $SL(n,Z)$ is same as $P(D+R)Q$ is in $SL(n,Z)$, thus we can replace $D$ with $PDQ$, so we can consider $PDQ$ to be the the follwoing matrix: the entries right above the diagonal are $d_1,d_2,\cdots,d_{n-1}$, the $(n,1)$ entriesentry is $d_n$, all other entries are $0$. Now let $PRQ$ be the identity matrix but only the $(n,1)$ entry being $-d_n$. This matrix $R$ is the one we are looking for.
In this argument, $D$ is not necessarily a diagonal matrix, actually $D$ can be any integer matrix, because we can use SNF to transform it to a diagonal matrix.
Conclusion: if $n>1$, any $n$ by $n$ integer matrix is the difference of a pair matrices in $Sl(n,Z)$.