Suppose $X$ and $Y$ are compact Hausdorff spaces. Let $\varphi\colon C(X)\to M_{n}(C(Y))$ be any $*$-homomorphism. If $\pi$ is an irreducible representation of $M_{n}(C(Y))$, then $\pi$ is unitarily equivalent to a point evaluation $\textrm{ev}_{y}$. The $*$-homomorphism $\textrm{ev}_{y}\circ\varphi\colon C(X)\to M_{n}(\mathbb{C})$ is a representation of $C(X)$. Since it's a finite-dimensional representation, we can find a unitary $u_{y}\in M_{n}(\mathbb{C})$ and a set of points $X_{y}=\{x^{y}_{1},\ldots,x^{y}_{n}\}\subset X$ such that for all $f\in C(X)$, $$ (\varphi\circ f)(y)=(\textrm{ev}_{y}\circ\varphi)(f)=u_{y} \begin{pmatrix} f(x^{y}_{1}) & 0 & \cdots & 0\\ 0 & f(x^{y}_{2}) & \cdots & 0\\ \vdots & \vdots & \ddots & \vdots\\ 0 & 0 & \cdots & f(x^{y}_{n}) \end{pmatrix} u_{y}^{*}. $$

My question is:

Is the set $\widetilde{X}:=\bigcup_{y\in Y}X_{y}$ closed in $X$?

This question is in a similar vein to one of my earlier questions: Closeness of points in the irreducible decomposition of a C$^{*}$-algebra representation