This answer is incomplete but I'll try to finish it later.

First note that you can't express the answer only using $Y$, even in the $g=1$ case. If the imaginary part of $\tau$ goes to $0$, you do not know anytihng about what $j(\tau)$ is doing.

I think you want to work with the induced quadratic form on pairs $(v,w) \in \mathbb Z^2 \times \mathbb Z^2 = \mathbb Z^4$ given by

$$ (v_1,w_1) \cdot (v_2,w_2) = \operatorname{Re}\left( \overline{(v_1 + Z w_1)}^T Y^{-1}(v_2 + Z w_2) \right)$$

If I calculated correctly, this is the actual Riemann form of your torus, because it comes from a Hermitian form $Y^{-1}$ on $\mathbb C^2$ whose imaginary part resticts to the standard symplectic from on $\mathbb Z^4$.

The action of $SP_{4}(\mathbb Z)$ on this is the natural action on quadratic forms in four variables. Indeed, $v^T Y^{-1} v$ on $\mathbb C^2$ is the unique Hermitian form whose imaginary part restricts to the standard symplectic form on the lattice $\mathbb Z^4$ embedded by the map $(v_1, v_2) \mapsto v_1 + Z v_2$. The action of $SP_{4}$ on $Z$ comes from the action of $SP_4$ on this lattice plus some weird action on $\mathbb C^2$, but since the action preserves the standard symplectic form it will preserve the uniquely determined Hermitian form.

Now I think a basis of neighborhoods of the rank $k$ component of the boundary of this lattice consists of points where there are $k$ vectors $v_1,\dots,v_k$ in $\mathbb Z^4$ that are linearly independent, whose pairing under the symplectic forms with each other all vanish, and whose lengths under the quadratic form are all $<\epsilon$.

I think this because if you have a sequence of such quadratic forms with vectors $v_1,\dots,v_k$ so that the length of those vectors goes to $0$, you obtain a lattice of rank $4-2k$ with a symplectic form by modding out by $v_1,\dots,v_k$ and taking the kernel of the map to $v_1,\dots, v_k$, and you obtain a corresponding quadratic form by taking the limit of the original quadratic form, which will be well-defined as the lengths of $v_1,\dots,v_k$ goes to $0$. So a sequence of points that are in all the rank $k$ neighborhoods but not all the rank $k+1$ neighborhoods will have an accumulation point in $\mathcal A^{2-k}$. That is precisely what the Satake compactification is supposed to look like. I will try to rigorously justify this...

In fact the condition that $v_1,\dots,v_k$ have zero pairing under the symplectic form is irrelevant. As the symplectic form and quadratic form come from the same Hermitian forrm, they are related by a Cauchy-Schwartz inequality, so if $v_1,v_2$ have length at most $\epsilon$, their pairing is at most $\epsilon^2$, so because the symplectic form is integral it is $0$. So we can express the condition only in terms of tuples of short vectors in the quadratic form with matrix

$$\begin{pmatrix} Y^{-1} & Y^{-1} X \\ X Y^{-1} & XY^{-1}X + Y\end{pmatrix}$$

That's totally explicit but might not be too fun. An alternate desciption of the system of neighborhoods of the rank $2$ part which may or may not be easier is the $SP_4(\mathbb Z)$ orbit of the set of pairs $X,Y$ where the shortest vector of $Y$ has length at least $1/\epsilon$. This is the same because you can always transform your two short vectors to be the first two basis vectors.