You actually care about the number of *singular* matrices (which is the difference between the number of invertible matrices and the number of unrestricted matrices). This has been studied: see 

Author Yonatan R. Katznelson
Title: Integral Matrices of Fixed Rank
Journal: proceedings of the AMS, 120(3) 1994