I think I figured it out myself. What was meant is that for every finite subset $S$ of $A_{\delta}$ one has
$$| \lbrace (n,m) \in S \times S \mid n-m \in A_{\delta^2/2} \rbrace | \geq \frac{\delta^2 |S|^2}{2}.$$
This follows from the proof of the second part of Lemma 4.37 in *Tao and Vu, Additive Combinatorics*.