Be aware that this is a somewhat vague comment by Saito which can be interpreted in numerous ways. I would not take what he has written too seriously.
I would guess that he means that $$\mathrm{Gal}(\bar{\mathbb{Q}}/\mathbb{Q}) = \lim_U \,\pi_1(U)$$ where the limit is over all open subsets $U \subset \mathrm{Spec} \mathbb{Z} = \{2,3,5, \ldots \}$. (Note that you have misquoted him, as he does indeed talk about open subsets of the set of all primes). Saito is being a bit naughty as $\mathrm{Spec} \mathbb{Z}$ is more than just a set, as it also comes equipped algebro-geometric structure given by its structure sheaf, which is obviously crucial.
The fundamental group $\pi_1(U)$ is the Galois group of the maximal algebraic field extension of $\mathbb{Q}$ which is unramified at all the primes in $U$. You can also throw in the infinite place if you like to account for ramification at $\infty$.