The conjecture is true. By the known factorization of Chebyshev polynomials, $$F_p(x)=\prod_{\substack{1\leq m\leq 2p-1\\(m,4p)=1}}(x-\zeta^m-\zeta^{-m}),$$ where $\zeta\in\mathbb{C}$ is a primitive $4p$-th root of unity. The splitting field of $F_p(x)$ is the subfield of the $4p$-th cyclotomic field fixed by complex conjugation; it is of degree $p-1$.
Assume that $q\nmid 2p$ is a prime number such that the reduction of $F_p(x)$ mod $q$ has a root in $\mathbb{F}_q$. The roots of $F_p(x)$ in $\overline{\mathbb{F}_q}$ are of the form $\xi^m+\xi^{-m}$, where $\xi\in\overline{\mathbb{F}_q}$ is a primitive $4p$-th root of unity. By assumption, the Frobenius automorphism $t\mapsto t^q$ fixes one of these roots, which is only possible when $q\equiv\pm 1\pmod{4p}$. It follows that, for any $a\in\mathbb{Z}$, the prime factors of $F_p(a)$ coprime to $2p$ are congruent to $\pm 1$ modulo $4p$. In particular, if $4p+1$ divides $F_p(a)$, then the only prime factor of $4p+1$ can be itself, i.e., $4p+1$ is prime.