A sum-of-entries version of Specht's theorem was proven in [Theorem 4.3 of Grohe et al. (2021): Homomorphism Tensors and Linear Equations][1]. The condition you're describing is also sufficient. More precisely, for matrices $A$ and $B$ it holds that $\sigma(w(A,A^*)) = \sigma(w(B,B^*))$ if and only if there exists a pseudo-stochastic matrix $X$ such that $XA = BX$ and $XA^* = B^*X$. A matrix is called *pseudo-stochastic* if $X \boldsymbol{1} = \boldsymbol{1} = X^T \boldsymbol{1}$ for $\boldsymbol{1}$ the all-ones vector.


  [1]: https://arxiv.org/abs/2111.11313