It suffices that there is a straight line (identified with $\Bbb R$ below) in $\Bbb R^4$ so that the connection matrix $A$ is $L^1$ (with respect to the trivial connection) when restricted to that line, and so that the gauge transformation $\sigma$ is $L^1_1$ (meaning that $\sigma: \Bbb R^4 \xrightarrow{a.e.} G \subset \Bbb R^N$ for some large $N$ has $\sigma - 1 \in L^1(\Bbb R^4, \Bbb R^N)$ and $\nabla \sigma \in L^1$). I have not made any effort to ensure that $\mathcal G_{decay}$ is a Banach Lie group acting smoothly on $\mathcal A_{decay}$.
Given any smooth connection $A$ on any principal bundle, the action of the gauge transformation $\sigma$ on $A$ is $\sigma(A) = A - (d_A \sigma) \sigma^{-1}$. So the stabilizer of $A$ inside the gauge group of all smooth gauge transformations is the set of $A$-parallel gauge transformations. So what you are asking for is a decay condition on $A$ that guarantees that any $A$-parallel gauge transformation cannot decay at infinity; this is reasonable, since the notion of being parallel means that your section is constant "with respect to $A$".
To do this, recall that the holonomy $H(t) = \text{Hol}_{0 \to t}(A) \in G$ of a connection along an interval $I = [s_1,s_2]$ with $-\infty \leq s_1 < 0 < s_2 \leq \infty$ defines a continuous map $L^1(I; \mathfrak g) \to C^0(I;G)$, where in the case that one of the $s_i = \pm \infty$, I am claiming that $H(\pm \infty) = \lim_{t \to \pm \infty} H(t)$. This follows from the claim that just the map $\text{Hol}_{0 \to s_2}: L^1(I;\mathfrak g) \to G$ is continuous in $L^1$, as for any $L^1$ function $f$, subintervals of $I$ contained in $(N,\infty)$ for $N$ sufficiently large will have $\|f\|_{L^1}$ arbitrarily small.
As a consequence, one sees that we have two continuous maps $\text{Hol}_{\pm\infty}: L^1(\Bbb R;\mathfrak g) \to G$.
Next recall that if $\sigma$ is an $A$-parallel gauge transformation on $\Bbb R$, this means that $\sigma(t) = H(t) \sigma(0) H(t)^{-1}$. Thus, if $\sigma(\infty) = 1$, we see that $\sigma(t) = 1$ for all $t$.
This implies that if you demand your gauge transformations decay to the identity at infinity on this line --- which is guaranteed by demanding that the gauge transformations are in $L^1_{1}$ --- then they must be constant on our line. But parallel gauge transformations are globally constant (so long as $A$ is locally $L^1$ on every line, which is guaranteed if it's bounded), and so a gauge transformation that decays to the identity at infinity is constant.