I wonder about the reason to consider $$A_j=\sum_{k=0}^{p-1}\frac{k^j}{k+1}\binom{2k}{k}$$ instead of $$\tilde A_j = \sum_{k=0}^{p-1} k^j \binom{2k}{k},$$ as $A_j\pm A_0$ ($+$ for odd $j$ and $-$ for even $j$) are reduced to linear combinations of the latter. Let me introduce the functions $$F_j(x)=\left(x\frac{d}{dx}\right)^j\sum_{k=0}^{p-1}\binom{2k}{k}x^k,$$ so that $\tilde A_j=F_j(1)$. Note that $$F_0(x)=F(x)=\sum_{k=0}^{p-1}\binom{2k}{k}x^k$$ satisfies the differential equation $$F(x)-\frac12(1-4x)F'(x)=\frac{(2p-1)!}{(p-1)!^2}x^{p-1}.$$ Here the coefficient on the right is $$\frac{(2p-1)!}{(p-1)!^2}\equiv p\pmod{p^4}$$ by Wolstenholme's theorem and we can write the equation in the form $$xF_0(x)-\frac12(1-4x)F_1(x)=\frac{(2p-1)!}{(p-1)!^2}x^p.$$ Applying repeatedly the operator $x\dfrac{d}{dx}$ to both sides of the identity produces, for each $j$, a relation of the form $$P_{0,j}(x)F_0(x)+P_{1,j}(x)F_1(x)+\dots+P_{j-1,j}(x)F_{j-1}(x)-\frac12(1-4x)F_j(x)=p^{j-1}\frac{(2p-1)!}{(p-1)!^2}x^p,$$ where all coefficients of the polynomials $P_{j,i}(x)$ P_{i,j}(x)$are integral (and, of course, independent of$p$). This gives a way to express$\frac23F_j(1)=\frac23\tilde \frac32F_j(1)=\frac32\tilde A_j$as a$\mathbb Z$-linear combination of$\tilde A_0,\tilde A_1,\dots,\tilde A_{j-1}$independently of$p$modulo$p^j$. The remaining piece is to check that$\tilde A_0$and$\tilde A_1$as well as the starting$A_0$do satisfy the "$p$-independence" property modulo$p^2$. The construction above gives a recipe to construct recursively the sequence$\tilde A_j$(hence$A_j$), although I do not try myself to figure out a closed-form evaluation of the auxilliary polynomials$P_{i,j}(x)$and/or their values at$x=1$. 1 I wonder about the reason to consider $$A_j=\sum_{k=0}^{p-1}\frac{k^j}{k+1}\binom{2k}{k}$$ instead of $$\tilde A_j = \sum_{k=0}^{p-1} k^j \binom{2k}{k},$$ as$A_j\pm A_0$($+$for odd$j$and$-$for even$j$) are reduced to linear combinations of the latter. Let me introduce the functions $$F_j(x)=\left(x\frac{d}{dx}\right)^j\sum_{k=0}^{p-1}\binom{2k}{k}x^k,$$ so that$\tilde A_j=F_j(1)$. Note that $$F_0(x)=F(x)=\sum_{k=0}^{p-1}\binom{2k}{k}x^k$$ satisfies the differential equation $$F(x)-\frac12(1-4x)F'(x)=\frac{(2p-1)!}{(p-1)!^2}x^{p-1}.$$ Here the coefficient on the right is $$\frac{(2p-1)!}{(p-1)!^2}\equiv p\pmod{p^4}$$ by Wolstenholme's theorem and we can write the equation in the form $$xF_0(x)-\frac12(1-4x)F_1(x)=\frac{(2p-1)!}{(p-1)!^2}x^p.$$ Applying repeatedly the operator$x\dfrac{d}{dx}$to both sides of the identity produces, for each$j$, a relation of the form $$P_{0,j}(x)F_0(x)+P_{1,j}(x)F_1(x)+\dots+P_{j-1,j}(x)F_{j-1}(x)-\frac12(1-4x)F_j(x)=p^{j-1}\frac{(2p-1)!}{(p-1)!^2}x^p,$$ where all coefficients of the polynomials$P_{j,i}(x)$are integral (and, of course, independent of$p$). This gives a way to express$\frac23F_j(1)=\frac23\tilde A_j$as a$\mathbb Z$-linear combination of$\tilde A_0,\tilde A_1,\dots,\tilde A_{j-1}$independently of$p$modulo$p^j$. The remaining piece is to check$\tilde A_0$and$\tilde A_1$modulo$p^2\$.