One way to define toric varieties is as quotients of affine $n-$space by the action of some torus. However, this is not strictly true as we need to throw away "bad points" which ruin this construction.

For example consider the construction of projective space as a toric variety. Let $\mathbb{G}_m$ act on $\mathbb{A}^n$ in the obvious way. Then the quotient of $\mathbb{A}^n$ by this action is a single point, as "everything is rescaled to the origin". More rigously the only functions invariant under this action are the constants, thus the quotient is the spectrum of the ground field. To fix this we of course we remove the origin and then take the quotient and we get projective space as required.

So given an action of some torus on affine space, how do we know which points to remove before we take the quotient to make sure we get a toric variety?

My first naive guess is to remove the points which are fixed under the action, but Im wary it may be more subtle than that, as I know GIT can get quite technical.