Let $T$ be a split torus over a field $F$.
The Weyl group is by definition a subgroup of the group of outer morphisms (defined over $F$).
There are only two automorphism (the inverse and the identity) of $T$ (defined over F), so the group of outer automorphism is isomorphic to $\mathbb{Z}/2$.
Both can realized by the conugation via the non-trivial element $\begin{pmatrix} 0 & -1 \newline 1 & 0 \end{pmatrix}$ and by the identity elment respectively.
So the Weyl group has at least two elements, hence exactly two elements.
Similiar you can argue for the maximal split torus in $SL(n)$ or $GL(n)$.
For non-split tori, the question is more difficult to answer. I know only the $GL(n)$-situation. In $GL(2)$, the tori is isomorphic to a multiplicative group of a quadratic field extensions, and the outer automrophisms automorphisms of the tori are somorphic isomorphic to the Galois group. Now these can be realized by conjugation of elements $GL(2)$, but I am not sure whether they can not be realized by elements of determinant $1$.In fact, I would bet against it.
