I think adding a few assumptions is fair here, as I'd guess there are too many classes to talk about easily otherwise.

I'm going to assume both that our class is a vector space and that it includes its own powers. I'll save some discussion on those assumptions to the end.

If we assume that powers and sums are allowed, then any symmetric matrix can be expressed as a linear combination of commuting projections that it generates (projections to its eigenspaces). As such, our problem then becomes one of which sets of projections are allowed.

Your assumption includes the assumption that $tr((AB)^k) = tr(A^k B^k)$. If we assume $A, B$ are projections, this reduces to the assumption that $tr((AB)^k) = tr(AB)$ for all $k$. But this implies that the eigenvalues of $AB$ must be 0 or 1 - so $AB$ is also a projection. As stated at https://proofwiki.org/wiki/Product_of_Projections, this implies that $A, B$ commute - so we are looking for sets of commuting projections. Therefore, having a set of commuting symmetric projections is equivalent to your condition with my assumptions. If we assume the set of projections is maximal (which we can without loss of generality), then it must be conjugate to the set of diagonal matrices.

Assumption 2 (assuming the vector space part) is equivalent to changing the problem to allowing $A$ and $B$ to be to different powers, rather than both being $k$ - in other words, the question becomes:

Let $P_1, P_2, ..., P_{j+k}$ be $j$ copies of $A$ and $k$ copies of $B$ in arbitrary order. Then:

**Question.** What are "non-trivial" classes of symmetric matrices such that $\text{tr}(P_1 P_2 ... P_{j + k}) = \text{tr}(A^j B^k)$for any permutation of the $P_i$?

I think it may be possible to drop it by looking at the growth of $tr((AB)^k)$, but haven't worked out exactly how that would work.

Assumption 1 seems to be the big one here. I'm not sure how to stop relying on it directly, but it seems a natural choice here.