I discussed this problem with fedja (Overflow pen-name), and he explained to me that the answer is no. As fedja is apparently busy, I post the answer:-) One cannot improve the exponent $p=2$ in any dimension. The reason is that for every $p>2$ there exists a distribution on the line whose support has Lebsegue measure $0$ and whose Fourier transform belongs to $L^p$. Now, for arbitrary dimension, one simply takes a product. In Russian, this is called Inashev-Musatov theorem (1957) but this was an improvement of a series of earlier results; apparently the condition $p>2$ is due to Wiener, Amer. J. Math. 60 (1938).
This results were for Fourier series rather than Fourier transform, but here is the reference for Fourier transform: MR0227693.
So one needs stronger assumptions on support than just zero measure. I don't know what these assumptions could be.