Probably not, but a proof is hopeless. Ruzsa and Gyarmati have a preprint in which they construct such a subset of size something like $N/\log \log N$.
Ben Green
- 4.8k
- 2
- 37
- 35