As suggested by Igor Rivin's argument (and following lines of argument suggested by Frobenius in the late 19th/early 20th century), it is the case that if we put the operator norm (with respect to Euclidean norm on vectors) on $M_{n}(\mathbb{C})$ and consider a finite subgroup $G$ of $U_{n}(\mathbb{C})$ such that $\|g^{2}-e \| < \frac{1}{6}$ for all $g \in G$, then $G$ is Abelian.

We may suppose that $G$ is irreducible, and primitive ( ie the given representation is not induced from a representation of a proper subgroup). We may and do, assume that $G$ is not nilpotent, since all irreducible complex representations of nilpotent groups are induced from $1$-dimensional representations, and the result is clear if $n = 1$. Since $G$ is primitive, every Abelian normal subgroup of $G$ consists of scalar matrices, so is central. We will obtain a contradiction by showing that $G^{\prime}$ is Abelian ( for then $G^{\prime} \leq Z(G)$ and $G$ is nilpotent).

Frobenius showed (actually with a different matrix norm, though the argument can be modified to work with the operator norm) that the elements $x \in G$ which satisfy $\| I - x \| < \frac{1}{2}$ generate an Abelian normal subgroup of $G$. We show that $\| ab - ba \| < \frac{1}{2}$ for all $a,b \in G$, so that $\|I - a^{-1}b^{-1}ab \| < \frac{1}{2}$ for all $a,b \in G$, and hence $G^{\prime}$ is Abelian.

Now $\|ab - ba \| \leq \|ab - ab^{-1}\| + \|ab^{-1} - a^{-1}b^{-1} \| + \|a^{-1}b^{-1} -ba \|$ = $\|b - b^{-1}\| + \|a - a^{-1} \| + \|(ba)^{-1} -ba \|$ = $\|b^{2} - I\| + \|a^{2} - I \| + \|(ba)^{2}-I \|$ < $\frac{1}{2}$. Thus $\| I - a^{-1}b^{-1}ab \| < \frac{1}{2}$ for all $a,b \in G$, so that $G^{\prime}$ is Abelian, the required contradiction.






The second part of the answer is somewhat tangential to the original question, and follows the direction suggested by Sean Eberhard's comment.

For a finite group $G$, it is the case that if more than $\sqrt{\frac{5}{8}} |G|$ elements $x \in G$ have $x^{2} = e$, then $G$ is Abelian. The dihedral group of order $8$ ( I mean the one with $8$ elements) - and direct products of it with elementary Abelian $2$-groups as large as you like-show that this can't be improved much as a general bound, since a dihedral group $D$ of order $8$ contains $6$ elements which square to the identity and $ 6 < \sqrt{\frac{5}{8}} |D| <7$ in that case.

This is because (as noted in the paper linked to in Sean Eberhard's comment, and also previously noted by Brauer and Fowler), the count of solutions to $x^{2} = e$ given using the Frobenius-Schur indicator leads easily to $\sqrt{\frac{5}{8}} |G| < \sqrt{k(G)}\sqrt{|G|}$ in the case under consideration, where $k(G)$ is the number of conjugacy classes of $G$. Hence $\frac{k(G)}{|G|} > \frac{5}{8}$, so the probability that two elements of $G$ commute is greater than $\frac{5}{8}$, in which case $G$ is Abelian by a Theorem of W.Gustafson. If preferred, this can be seen directly using character theory- in general, if $G$ has $k$ conjugacy classes and $Z = Z(G)$, we have $|Z| + 4(k - |Z|) \leq |G|$ by the orthogonality relations. Hence $\frac{k}{|G|} \leq \frac{1}{4} + \frac{3|Z|}{4|G|} \leq \frac{5}{8}$ if $Z < G$.