Given any two tracial von Neumann algebras $(N_1, \tau_1)$ and $(N_2, \tau_2)$ the $L^2$ space of the free product $(N_1 * N_2, \tau)$ canonically decomposes as
$$
L^2(N_1 * N_2, \tau) = \mathbb C \oplus_{n \in \mathbb N} \bigoplus_{i_1 \not= i_2, i_2 \not= i_3, \ldots, i_{n - 1} \not= i_n } \overline {\otimes}_{k = 1}^n L^2_0(N_{i_k}, \tau_{i_k}),
$$
where $L^2_0(N_i, \tau_i)$ is the orthogonal complement of the scalars.
From this decomposition it's not hard to see that as an $N_1$ bimodule, $L^2(N_1 * N_2, \tau)$ decomposes as a direct sum of one copy of the trivial bimodule $L^2(N_1, \tau_1)$ and copies of the coarse bimodule $L^2(N_1 \overline \otimes N_1, \tau_1 \otimes \tau_1)$. Specifically, if $\xi \in L^2(N_1 * N_2, \tau)$ is a unit vector which is an ``elementary word'' starting and ending with vectors in $L^2_0(N_2, \tau_2)$ then $x \xi y \mapsto x \otimes y$ extends to an isomorphism from the $N_1$ bimodule generated by $\xi$ and $L^2(N_1 \overline \otimes N_1, \tau_1 \otimes \tau_1)$.
Thus $N_1' \cap (N_1 * N_2) \not= \mathcal Z(N_1)$ if and only if the coarse $N_1$ bimodule has non-zero central vectors, which, by identifying $L^2(N_1 \overline \otimes N_1, \tau_1 \otimes \tau_1)$ with Hilbert-Schmidt operators and then taking a spectral projection, is if and only if $L^2(N_1, \tau_1)$ has a finite dimensional $N_1$-sub-bimodule, which is if and only if $N_1$ has a non-trivial finite dimensional direct summand.
In the case you're looking at $N_1$ is a ${\rm II}_1$ factor and so $N_1' \cap (N_1 * N_2) = \mathcal Z(N_1) = \mathbb C$.