Is there an example of a representation $\rho: G \rightarrow GL(V)$ for some finite group $G$ where say $W \subset V$ is a $G$-invariant subspace for $\rho$ but the orthogonal complement (in the standard sense) $W^{\perp}$ is *not* G-invariant? I understand one could "unitarize" the representation using Weyl's averaging trick (and getting a new inner product) but my question is to find an example where one cannot do away with this averaging (and use the usual inner product)

Try $G=\{1,-1\}$ and $\rho\colon G \to GL_2\mathbb{R}$ where $\rho(-1)$ is the matrix

$$\left(\begin{matrix}-1&2\\ 0&1\end{matrix}\right)$$

Take $W$ to be the span of

$$\left(\begin{matrix}1\\ 0\end{matrix}\right)$$

and use the standard inner product on $\mathbb{R}^2$.

Hurwitz’saveraging method then applies. $\endgroup$ – Francois Ziegler Jul 3 '20 at 13:10