2 added 492 characters in body

I learned the second congruence as a version of Wolsteholme's theorem, and I would be a bit surprised if Kazandzidis was the first person to observe the equivalence between this form and any other form of Wolsteholme's result. As for the "reason" that this result is true, I wrote a proof for the Wikipedia page which is mostly, but not entirely, a direct counting argument and which you could call "the grounds" for the congruence. The conceptual content of the argument is as follows:

The result holds modulo $p^2$ because you can divide the $pm$-set into $m$ cycles of length $p$ and rotate them separately. You obtain $\binom{m}{n}$ equivalence classes of subsets of size 1, and the other equivalence classes have order $p^2$ or higher.

Then you can examine $\binom{-p}{p}$, interpreted as $\binom{p^4-p}{p}$. This binomial coefficient is algebraically equivalent to $\binom{2p}{p}/2$ up to sign. The orbit decomposition in the previous paragraph establishes a second relation with $\binom{2p}{p}$. The conclusion is that the mod $p^2$ contribution vanishes for one non-trivial pair $(m,n)$, and if it vanishes once, it vanishes always. This vanishing principle extends mod $p^4$ — your second binomial congruence holds mod $p^4$ — provided that it vanishes once "by accident". A prime for which this happens is called a Wolstenholme prime. Two such primes are known, 16843 and 2124679.

Another remark: There are two pieces of evidence that the $p^2$ congruence (Babbage's theorem) is entirely combinatorial, but the extension to $p^3$ (Wolstenholme) is essentially algebraic. First, that Wolstenholme's theorem doesn't hold for the primes $p=2,3$. Second, that Babbage's theorem has a $q$-analogue for Gaussian binomial coefficients, with the same orbit proof. But Wolsteholme's extension does not have a $q$-analogue as far as I know. In fact, you can tell that the known $q$-Babbage's theorem doesn't extend, because the difference polynomial doesn't have any more cyclotomic factors.

I stand corrected on a couple of points. First, on the math, there is a paper by George Andrews, "q-Analogs of the binomial coefficient congruences of Babbage, Wolstenholme, and Glaisher", that does give some version of a q-analogue of Wolstenholme's theorem. The proof given there is algebraic, so it does not shed light on possible combinatorial "explanations". Second, according to Andrews, the full binomial interpretation of Wolstenholme's result is due to Glaisher.

1

I learned the second congruence as a version of Wolsteholme's theorem, and I would be a bit surprised if Kazandzidis was the first person to observe the equivalence between this form and any other form of Wolsteholme's result. As for the "reason" that this result is true, I wrote a proof for the Wikipedia page which is mostly, but not entirely, a direct counting argument and which you could call "the grounds" for the congruence. The conceptual content of the argument is as follows:

The result holds modulo $p^2$ because you can divide the $pm$-set into $m$ cycles of length $p$ and rotate them separately. You obtain $\binom{m}{n}$ equivalence classes of subsets of size 1, and the other equivalence classes have order $p^2$ or higher.

Then you can examine $\binom{-p}{p}$, interpreted as $\binom{p^4-p}{p}$. This binomial coefficient is algebraically equivalent to $\binom{2p}{p}/2$ up to sign. The orbit decomposition in the previous paragraph establishes a second relation with $\binom{2p}{p}$. The conclusion is that the mod $p^2$ contribution vanishes for one non-trivial pair $(m,n)$, and if it vanishes once, it vanishes always. This vanishing principle extends mod $p^4$ — your second binomial congruence holds mod $p^4$ — provided that it vanishes once "by accident". A prime for which this happens is called a Wolstenholme prime. Two such primes are known, 16843 and 2124679.

Another remark: There are two pieces of evidence that the $p^2$ congruence (Babbage's theorem) is entirely combinatorial, but the extension to $p^3$ (Wolstenholme) is essentially algebraic. First, that Wolstenholme's theorem doesn't hold for the primes $p=2,3$. Second, that Babbage's theorem has a $q$-analogue for Gaussian binomial coefficients, with the same orbit proof. But Wolsteholme's extension does not have a $q$-analogue as far as I know. In fact, you can tell that the known $q$-Babbage's theorem doesn't extend, because the difference polynomial doesn't have any more cyclotomic factors.