Here is a proof of "yes," using Tim Campion's proposition below.

Let $p$ be the smallest prime in $P$. For any prime $q\ne p$, let $o_q(p)$ be the multiplicative order of $p$ modulo $q$, or equivalently the least $n$ such that $q$ divides $|GL_n(p)|$. Assuming $P\ne\{p\}$, let $q\in P-\{p\}$ and $G=GL(V)$ where $V$ is a vector space of order $p^n$ and $n=o_q(p)$. As $p$ is smallest, $n\ge2$. By assumption, $G$ contains a Hall $P$-subgroup $X$. Then $X$ contains a Sylow $p$-subgroup $U$ of $G$ as well as an element $x$ of order $q$. If $U$ is normal in $X$, then $|X|$ divides $|U|(p-1)^n$, the order of the full upper triangular group, which is not divisible by $q$ as $q>p$. So $U$ is not normal in $X$. The theory of $B-N$ pairs then implies that $X$ contains a copy of $SL_2(p)$. Hence every prime divisor of $p-1$ lies in $P$, which forces $p=2$. Then $U$ is a Borel subgroup of $G$, so $X$ must be a parabolic subgroup of $G$. But because of $x$, $X$ stabilizes no proper subspace of $V$. The only such parabolic subgroup is $X=G$. Hence $P$ contains all prime divisors of $|G|$. In particular, $3\in P$.

Now suppose that $P$ is not the set of all primes and choose a prime $r\not\in P$ to minimize $m=o_r(2)$. Since $3\in P$, $m\ge3$. Let $H=GL(W)$, where $W$ is a vector space of order $2^m$. Then $H$ contains an element $y$ of order $r$. Let $U$ be a Sylow $2$-subgroup of $H$. Let $W_1$ and $W_{m-1}$ be $U$-invariant subspaces of $W$ of respective dimensions $1$ and $m-1$. Let $H_1$ and $H_{m-1}$ be their respective stabilizers in $H$. Then $H_1$ and $H_{m-1}$ are maximal parabolic subgroups of $H$ containing $U$, and each is an extension of an elementary abelian $2$-group by $GL_{m-1}(2)$. Hence $H_1$ and $H_{m-1}$ are $P$-groups, by our choice of $r$. However, they are maximal subgroups of $H$ and they are not conjugate in $H$, being distinct parabolic subgroups containing $U$.

By assumption, $\langle H_1^g, H_{m-1}\rangle$ must be a $P$-group for some $g\in H$. Since $H_{m-1}$ is maximal and not equal to $H_1^g$, $H$ must be a $P$-group. But $y\in H$ has order $r\not\in P$, contradiction.

$\DeclareMathOperator\GL{GL}\newcommand\card[1]{\lvert#1\rvert}$Here is a proof of "yes," using Tim Campion's proposition below.

Let $p$ be the smallest prime in $P$. For any prime $q\ne p$, let $o_q(p)$ be the multiplicative order of $p$ modulo $q$, or equivalently the least $n$ such that $q$ divides $\card{\GL_n(p)}$. Assuming $P\ne\{p\}$, let $q\in P-\{p\}$ and $G=\GL(V)$ where $V$ is a vector space of order $p^n$ and $n=o_q(p)$. As $p$ is smallest, $n\ge2$. By assumption, $G$ contains a Hall $P$-subgroup $X$. Then $X$ contains a Sylow $p$-subgroup $U$ of $G$ as well as an element $x$ of order $q$. If $U$ is normal in $X$, then $\card X$ divides $\card U(p-1)^n$, the order of the full upper triangular group, which is not divisible by $q$ as $q>p$. So $U$ is not normal in $X$. The theory of $B$-$N$ pairs then implies that $X$ contains a copy of $\operatorname{SL}_2(p)$. Hence every prime divisor of $p-1$ lies in $P$, which forces $p=2$. Then $U$ is a Borel subgroup of $G$, so $X$ must be a parabolic subgroup of $G$. But because of $x$, $X$ stabilizes no proper subspace of $V$. The only such parabolic subgroup is $X=G$. Hence $P$ contains all prime divisors of $\card G$. In particular, $3\in P$.

Now suppose that $P$ is not the set of all primes and choose a prime $r\not\in P$ to minimize $m=o_r(2)$. Since $3\in P$, $m\ge3$. Let $H=\GL(W)$, where $W$ is a vector space of order $2^m$. Then $H$ contains an element $y$ of order $r$. Let $U$ be a Sylow $2$-subgroup of $H$. Let $W_1$ and $W_{m-1}$ be $U$-invariant subspaces of $W$ of respective dimensions $1$ and $m-1$. Let $H_1$ and $H_{m-1}$ be their respective stabilizers in $H$. Then $H_1$ and $H_{m-1}$ are maximal parabolic subgroups of $H$ containing $U$, and each is an extension of an elementary abelian $2$-group by $\GL_{m-1}(2)$. Hence $H_1$ and $H_{m-1}$ are $P$-groups, by our choice of $r$. However, they are maximal subgroups of $H$ and they are not conjugate in $H$, being distinct parabolic subgroups containing $U$.

By assumption, $\langle H_1^g, H_{m-1}\rangle$ must be a $P$-group for some $g\in H$. Since $H_{m-1}$ is maximal and not equal to $H_1^g$, $H$ must be a $P$-group. But $y\in H$ has order $r\notin P$, contradiction.