Convex hull of the orbit of a matrix under permutations

Let $P$ be a generic permutation matrix on $\mathbb{R}^n$. For any vector $x \in \mathbb{R}^n$, the convex hull of the set $\{ Px : \; \text{$P$is a permutation matrix}\}$ is the set of vectors majorized by $x$. Now for a matrix $X \in \mathbb{R}^{n \times n}$ consider the set $$\{ P X P^T : \text{P is a permutation matrix}\}$$ Is there a similar characterization of the convex hull of the above set?