Suppose that $A_1, \dots , A_n, B_1, \dots , B_n \in \Bbb C^{d \times d}$ and that, for every $x \in \mathbb{C}^n$, the following holds $$\left( \sum_i x_i A_i \right) \left( \sum_i x_i A_i \right)^{\dagger} = \left( \sum_i x_i B_i \right) \left( \sum_i x_i B_i \right)^{\dagger} $$
where $\dagger$ denotes the adjoint (i.e., transpose conjugate) of a matrix. Is it true that there exists a unitary matrix $\Sigma \in {\rm U}(d, \mathbb{C})$ such that $A_i = \Sigma B_i$ for all $i$?