You actually care about the number of singular matrices (which is the difference between the number of invertible matrices and the number of unrestricted matrices). This has been studied: see
Author Yonatan R. Katznelson Title: Integral Matrices of Fixed Rank Journal: proceedings of the AMS, 120(3) 1994