One can take the continuum limit of your proof as $X \to \infty$, again using the prime number theorem, to obtain a proof that does not involve primes at all:

$$ \int f(t) \log \frac{1}{|t|}\ dt = \gamma - \sum_{\sigma = \pm 1} \int_0^\infty f(\sigma t) (\log t + \gamma)\ dt $$
$$ = \gamma - \lim_{\varepsilon \to 0} \sum_{\sigma = \pm 1} \int_0^\infty f(\sigma t) (\log \frac{t}{\varepsilon} + \gamma)\ dt  + \log \frac{1}{\varepsilon}$$
$$ = \gamma - \lim_{\varepsilon \to 0} \sum_{\sigma = \pm 1} \int_0^\infty f(\sigma t) (\sum_{0 < s < t/\varepsilon} \frac{1}{s})\ dt + \log \frac{1}{\varepsilon}$$
$$ = \gamma - \lim_{\varepsilon \to 0} \sum_{\sigma = \pm 1} \int_0^\infty f(\sigma t) (\sum_{0 < s < t/\varepsilon} \frac{1}{s})\ dt  + \log \frac{1}{\varepsilon}$$
$$ = \gamma - \lim_{\varepsilon \to 0} \sum_{\sigma = \pm 1} \sum_{s>0} \frac{1}{s} \int_{\varepsilon s}^\infty f(\sigma t)\ dt + \log \frac{1}{\varepsilon}$$
$$ = \gamma - \lim_{\varepsilon \to 0} \sum_{\sigma = \pm 1} \sum_{s>0} \int_{\varepsilon}^\infty f(\sigma s t)\ dt + \log \frac{1}{\varepsilon}$$
$$ = \gamma - \lim_{\varepsilon \to 0}  \int_{\varepsilon}^\infty \sum_{s \in \mathbb{Z} \backslash \{0\}} f(s t)\ dt + \log \frac{1}{\varepsilon}$$
$$ = \gamma - A - \lim_{\varepsilon \to 0}  \int_{\varepsilon}^1 \sum_{s \in \mathbb{Z} \backslash \{0\}} f(s t)\ dt + \log \frac{1}{\varepsilon}$$
$$ = \gamma - A - \lim_{\varepsilon \to 0}  \int_{\varepsilon}^1 (\sum_{s \in \mathbb{Z} \backslash \{0\}} f(s t) - \frac{1}{t})\ dt $$
$$ = \gamma - A - \lim_{\varepsilon \to 0}  \int_{\varepsilon}^1 (\frac{1}{t} \sum_{s \in \mathbb{Z} \backslash \{0\}} \hat f(s/t) - 1)\ dt $$
$$ = \gamma + 1 - A - B$$
where
$$ A := \int_1^\infty \sum_{s \in \mathbb{Z} \backslash \{0\}} f(st)\ dt$$
$$ = \int_{|t| \geq 1} f(t) (\sum_{1 \leq s \leq |t|} \frac{1}{s})\ dt$$
and
$$ B := \int_0^1 \sum_{s \in \mathbb{Z} \backslash \{0\}} \hat f(s/t)\ \frac{dt}{t}$$
$$ = \int_1^\infty \sum_{s \in \mathbb{Z} \backslash \{0\}} \hat f(st)\ \frac{dt}{t}$$
$$ = \int_1^\infty \hat f(t) \frac{\lfloor |t| \rfloor}{dt}\ dt.$$
Since $A,B$ are clearly non-negative, this gives your inequality.  This also shows that one is within $o(1)$ of equality if and only if one simultaneously has
$$ \int_{|t| \geq 1} f(t) (1 + \log |t|)\ dt = o(1)$$
and
$$ \int_{|t| \geq 1} \hat f(t)\ dt = o(1).$$
By the Hahn-Banach theorem, these estimates are incompatible with the hypotheses $f(0)=\hat f(0)=1$, $f(t), \hat f(t) \geq 0$ for $|t| \geq 1$ if and only if there exist non-negative measurable functions $a(t), b(t)$ supported on $|t| \geq 1$ with $\sup_t \frac{a(t)}{1+\log |t|}, \sup_t b(t) < \infty$ and numbers $\alpha,\beta$ not summing to zero, such that
$$ \alpha f(0) + \beta \hat f(0) = \int_{\mathbb R} f(t) a(t)\ dt + \int_{\mathbb R} \hat f(t) b(t)\ dt $$
for all Schwartz $f$, or equivalently that
$$ \alpha \delta + \beta = a + \check b$$
in the sense of tempered distributions, where $\delta$ is the Dirac delta.  But the right-hand side is continuous at the origin, so $\alpha$ must vanish; the Fourier transform of the right-hand side has a continuous antiderivative at the origin, so $\beta$ must vanish, contradiction.  This shows that one can make $A$ and $B$ simultaneously $o(1)$, so $1+\gamma$ is in fact optimal.  (But the invocation of the Hahn-Banach theorem makes it difficult to explicitly construct $f$ that come close to equality!)