I have a probabilistic approach showing your lower bound on $\alpha$ is sharp, and in fact your result can't be improved by more than a logarithmic factor, using the geometry of numbers.

The strategy is to count matrices with bounded entries of rank $\leq r$ (where $r=k-1$) and subtract off the ones which are not invertible. To count the non-invertible ones, we look at the lattice generated by their columns, which is a lattice of rank $m<n$, and then we prove an *upper* bound for the counting problem with the columns of the matrix restricted to lie in a lattice.

In fact, when counting matrices with rank $\leq r$, it is better to count them with a weight function that takes value $1$ on rank $r$ and larger values on ranks strictly smaller than $r$, as this weight function allows simple counts using geometry of numbers methods.

We begin by proving an upper bound in the lattice setting, and then, by similar methods, a lower bound. Using the lower bound for $\mathbb Z^n$ and then subtracting off the upper bound for sublattices, we will show that there are many such matrices with entries $< C p^{ \frac{n-r}{n}} \log p$.

**Lemma 1:** Let $\Lambda$ be a rank $m$ lattice and $R$ a positive real number. The number of tuples $v_1,\dots, v_n \in \Lambda$ with $|v_i| <R$ such that $v_1,\dots, v_n$ generate a subspace of rank $r$ of $\Lambda/p\Lambda$ and a sublattice of rank $m$ of $\Lambda$ is $$\ll\frac{R^{mn}}{ \operatorname{vol}(\Lambda)^n p^{ (m-r)(n-r) }} . $$

Proof: For each such tuple, there exists at least $1$ subspace $W$ of $\Lambda/ p \Lambda$ of dimension $r$ such that $v_1,\dots, v_n \in W$. Let's estimate the number of allowable tuples $v_1,\dots, v_n$ for a fixed $W$. Let $\Lambda'$ be the sublattice of $\Lambda$ consisting of vectors that modulo $p$ lie in $W$. Then $\Lambda'$ has volume $\operatorname{vol}(\Lambda) p^{ (m-r)}$. For any such tuple to exist, the $m$ successive minima of $\Lambda'$ must be $<R$. For a lattice $\Lambda'$ with all successive minima $<R$, the number of vectors of length $<R$ in the lattice is $\ll R^m / \operatorname{vol}(\Lambda')$. So the number of $n$-tuples is $$\ll\frac{ R^{mn} }{ \operatorname{vol}(\Lambda')^n} =\frac{ R^{mn} }{\operatorname{vol}(\Lambda)^n p^{ n (m-r)} } .$$ Summing over the $\ll p^{r (m-r)}$ choices of $W$, we get the claim.

A similar geometry of numbers approach can be used to get a lower bound. However, the lower bound is not quite for the quantity we want.

**Lemma 2:** Let $\Lambda$ be a rank $m$ lattice and $R$ a positive real number. The sum over tuples $v_1,\dots, v_n \in \Lambda$ with $|v_i| <R$ of the number of $r$-dimensional subspaces of $\Lambda / p\Lambda$ containing $v_1,\dots, v_n$ is $$ \gg \frac{R^{mn}}{ \operatorname{vol}(\Lambda)^n p^{ (m-r)(n-r) }} . $$

Proof: Exchanging the order of summation, this is the sum over subspaces $W$ of $\Lambda/p\Lambda$ of dimension $r$ of the number of $n$-tuples of vectors of length $<R$ in $\Lambda$ congruent mod $p$ to elements of $W$. Let $\Lambda'$ be the sublattice of $\Lambda$ consisting of vectors that modulo $p$ lie in $W$, with volume $\operatorname{vol}(\Lambda)/p^{n-r}$. We now use the fact that, regardless of the successive minima, the number of vectors of length $<R$ in the lattice is $\gg R^n / \operatorname{vol}(\Lambda')$ (Blichfeldt's theorem). Since the number of possible $W$ is $\gg p^{ r(m-r)}$, this gives the claim.

Specializing Lemma 2 to the integer lattice with $m=n$, we see that the sum over $n \times n$ integer matrices with column norms $<R$ of the number of rank $r$ subspaces of $\mathbb F_p^n$ containing the image mod $p$ is $ \gg \frac{R^{n^2}}{ p^{(n-r)^2}}$. This sum is supported on matrices of rank $\leq r$ mod $p$, so to show there exists an invertible matrix of rank $\leq r$ mod $p$, it suffices to show that the restriction of this sum to invertible matrices is nonzero. That is, it suffices to show

$$ \sum_{m=0}^{n-1} \sum_{ \substack{v_1,\dots, v_n \in \mathbb Z^m \\ \operatorname{rank} \langle v_1,\dots,v_n \rangle=m}} \# \{ W \subset \mathbb F_p^n \mid \dim W =r, v_i \mod p \in W\} = o \left( \frac{R^{n^2}}{ p^{(n-r)^2}} \right) $$
as then the contribution of the non-invertible matrices would be smaller than the total contribution and so the sum over invertible matrices is nonzero. We have (explanations to be offered after the chain of inequalities)

$$ \sum_{m=0}^{n-1} \sum_{ \substack{v_1,\dots, v_n \in \mathbb Z^m \\ \operatorname{rank} \langle v_1,\dots,v_n \rangle=m}} \# \{ W \subset \mathbb F_p^n \mid \dim W =r, v_i \mod p \in W\} $$

$$ \ll \sum_{m=0}^{n-1} \sum_{s=0}^{\min(m,r)} \sum_{ \substack{v_1,\dots, v_n \in \mathbb Z^m \\ \operatorname{rank} \langle v_1,\dots,v_n \rangle=m \\ \dim \langle v_1,\dots,v_n \mod p \rangle=s }} p^{ (m-r) (r-s)} $$

$$ = \sum_{m=0}^{n-1} \sum_{s=0}^{\min(m,r)} \sum_{\substack{ \Lambda \subset \mathbb Z^n \\ \operatorname{rank} \Lambda = m \\ \Lambda \textrm{ primitive}\\ \operatorname{vol}(\Lambda) \leq R^m}} \sum_{ \substack{v_1,\dots, v_n \in \Lambda \\ \operatorname{rank} \langle v_1,\dots,v_n \rangle=m \\ \dim \langle v_1,\dots,v_n \mod p \rangle=s }} p^{ (m-r) (r-s)} $$

$$ \ll \sum_{m=0}^{n-1} \sum_{s=0}^{\min(m,r)} \sum_{\substack{ \Lambda \subset \mathbb Z^n \\ \operatorname{rank} \Lambda = m \\ \Lambda \textrm{ primitive}\\ \operatorname{vol}(\Lambda) \leq R^m}} \frac{R^{mn}}{ \operatorname{vol}(\Lambda)^n p^{(m-s)(n-s)}} p^{ (m-r) (r-s)}$$

$$\ll \sum_{m=0}^{n-1} \sum_{s=0}^{\min(m,r)} \frac{R^{mn} \log R}{ p^{(m-s)(n-s)}} p^{ (m-r) (r-s)} $$
where the first inequality uses the fact that that the number of subspaces containing a given subspace is the number of $\mathbb F_p$-points on a Grassmanian, which is $\ll p$ raised to the dimension of that Grassmanian, the second equality uses the fact that $\langle v_1,\dots, v_n\rangle$ is contained in a unique primitive lattice $\Lambda$ of the same rank, and since $\Lambda$ is primitive, $\Lambda / p \Lambda \to \mathbb Z^n/p \mathbb Z^n$ is injective, so $v_1,\dots, v_n$ generate a subspace of the same rank in each, the next line uses Lemma 1, and the last inequality uses Schmidts theorem that the number of primitive lattices in $\mathbb Z^n$ with volume $<X$ is $\ll X^n$, together with an integration by parts.

We now investigate the sums in $m$ and $s$ in the final bound. The dependence on $m$ is given by $(R^n p^{r-s - (n-s)})^m$ or $(R^n p^{r-n})^m$. If $R > p^{\frac{n-r}{n}}$ then this is maximized at $m=n-1$, where we get a term of

$$\sum_{s=0}^r R ^{n(n-1)} p^{ (n-1-r)(r-s) - (n-s) (n-1-s) } \log R.$$

Differentiating the exponent with respect to $s$, we see it is maximized at $s =\frac{n+r}{2}$, which means on the interval $[0,r]$ it is maximized at $r$, meaning the sum is $\ll R^{ n(n-1)} p^{ - (n-r)(n-1-r)}\log R$, which, divided by the main term $R^{n^2} p^{-(n-r)^2}$, is $ p^{n-r} \log R / R^n$. Assuming $R$ is greater than $p^{ (n-r)/n}$ by a logarithmic factor, there are matrices with row norms $<R$ which are invertible and have rank $\leq r$ modulo $p$.