Suppose $X\subset \mathrm{GL}_n(p)$ is a set of invertible matrices such that for every $A,B\in X$ then also $A-B\in \mathrm{GL}_n(p)\cup \{0\}$. (If anyone knows a name for such sets I would be grateful).

I was trying to figure out what is the size of the largest such $X$. Since $\mathbb{F}_{p^n}$ can be embedded in $\mathrm{GL}_n(p)\cup \{0\}$ then obviously the size is at least $p^n-1$.

My interest in this question was motivated from a computer science question (ideal secret sharing schemes), and from there it appears that $p^n-1$ must also be the upper bound.

I was wondering if anyone knows a more direct way of showing that this holds (something more algebraic or direct combinatorial arguments).