To start with: this functional equation is a "Fredholm integral equation of second kind".
We use the Mellin transform to find solutions.
(See page 657 of "Handbook of integral equations").
Lets make Mellin transform on both side:
$$\int_0^\infty \int_0^\infty f(t) e^{-itx} \, dt \, x^{s-1} dx=\lambda \int_0^\infty \frac{1}{x} f\left(\frac{1}{x}\right) x^{s-1} dx$$
$$\int_0^\infty \int_0^\infty f(t) t^{-s}e^{-ix} \, x^{s-1} \, dt \,dx=\lambda \int_0^\infty f\left(x\right) x^{-s} dx$$
We note $\mathcal{M}(f)(s)$ the Mellin transform of $f$
$$\mathcal{M}(e^{-ix})(s)\cdot \mathcal{M}(f)(1-s) =\lambda \,\mathcal{M}(f)(1-s)$$
$$[\mathcal{M}(e^{-ix})(s) - \lambda] \mathcal{M}(f)(1-s)=0$$
$$\left[\cos\left(\frac{\pi s}{2}\right) + i\sin\left(\frac{\pi s}{2}\right) - \lambda\right] \mathcal{M}(f)(1-s)=0$$
Under this form, we see either $\mathcal{M}(f)(1-s)=0$ either we have $\lambda$ such that there exist $\alpha$ such that $\cos(\frac{\pi \alpha}{2}) + i\sin(\frac{\pi \alpha}{2})=\lambda$ and $\mathcal{M}(f)(1-s)=\delta(s-\alpha)$. So taking Mellin inverse of $\delta(s-\alpha)$ we see the functional equation has only solutions of the form $x^{a}$.