Combinations of positive semidefinite matrices

Let $U$ be some convex subset of $\mathbb{C}^d$, and define the set of square matrices $\mathcal{U} \subset \mathbb{C}^{d \times d}$ to be the set of all convex combinations of $x y^*$ such that $x \in U, y \in U$. (Here $^*$ denotes the conjugate transpose.)

Is it always true that if a matrix belongs to $\mathcal{U}$ and is Hermitian and positive semidefinite, then it can be written as a convex combination of terms $x x^*$ with $x \in U$?

Edit: The conjecture was shown to be false in the most general case in the answer below, but is it possible to formalise the constraints under which it is true?

This is a problem I've encountered when trying to generalize the property that each positive semidefinite matrix with trace 1 can be written as a convex combination of rank-one positive semidefinite terms $x x^*$ with trace 1. I was wondering how general it is and I did not manage to prove this.

No. If $U$ is a segment between $x$ and $y=ix$, then zero matrix equals $(xy^*+yx^*)/2$ and thus belongs to $\mathcal U$. Well, it is not positive definite, but only non-negative definite. This may be easily fixed by considering a neighborhood of the segment $U$ (instead of $U$) and taking a convex combination of zero and other matrices, which is still too close to zero.