This is not a complete answer but only deals with the rank 2 case with trivial determinant:
Let $u=\Phi\neq0$ be a nilpotent Higgs field, then $\Phi^2=0,$ so the kernel
line bundle $L\subset E$ (assuming $\Phi$ is not identically zero) contains the image of $\Phi.$
Hence, $\Phi$ is uniquely
determined by a holomorphic section $\phi\in H^0(X;L^2K).$
This shows that a sufficient condition of a holomorphic rank 2 bundle with trivial determinant is that it does not contain any holomorphic line bundle
$L$ such that $H^0(X;L^2K)\neq\{0\}.$

In particular, in the case of a surface of genus 2, stable holomorphic bundles of rank 2 with trivial determinant are uniquely determined by the divisor of holomorphic line subbundles in $Pic_{-1}(X)$ by Narasimhan-Ramanan. Therefore, in this situation the aforementioned condition is very natural. In particular it also follows from Narasimhan-Ramanan that the set of very-stable rank 2 bundles with trivial determinant over a genus 2 surface is $\mathbb CP^3$ without the Kummer surface (of $X$, corresponding to strictly semi-stable bundles) and without several copies of $\mathbb CP^2$ corresponding to the space of non-trivial extensions
$$0\to S^*\to E\to S\to 0$$
of the 16 spin bundles $S$ on $X.$