$Hom_G(V^*,V) \cong Hom(V^*,V)^G \cong (V\otimes V)^G \cong (Sym^2 V\oplus Alt^2 V)^G$, so the first is nonzero ($V$ is self-dual) iff $V$ possesses a symmetric or alternating invariant form. By Schur's lemma it can have only one of the two.
It's not as easy to do in your head, but you can indeed compute the weight multiplicities of $Alt^2 V$ from those of $V$: $m_\nu(Alt^2 V) = \sum_{\{\lambda,\mu\}, \lambda\neq \mu} m_\lambda(V) m_\mu(V)$.
Now you need to determine whether those multiplicities give a representation with an invariant vector. This is the most annoying part. For each positive root $\beta$, we have a differencing operator "at $\mu$, subtract the value located at $\mu+\beta$", and these commute. Apply all of them and see if the value at the origin is $1$ or $0$. (This is undoing the denominator in the Weyl character formula, just Fourier transformed to deal with weight multiplicities.) If $1$, then $V$ is quaternionic (for $G$ compact); if $0$, then $V$ is real (for $G$ compact, and $V$ assumed self-dual).
The latter half of this answer may not be so different from answering your boldface question with "Yes, in principle you must be able to, since the representation is characterized by its multiplicity diagram."