The $N\times N$ random matrices with real or complex entries are generalizations of nonhermitian gaussian ensembles, also known as Girko ensemble: the entries are independent and identically distributed, but with arbitrary, not necessarily Gaussian, distribution. Whereas nonhermitian gaussian ensembles are unitarily invariant this is not true any more in the general case; thus we cannot use the results about Haar unitary random matrices to derive asymptotic freeness results for the nonhermitian and non gaussian ensembles. Nevertheless, there are many results in the literature which show that the $N\times N$ random matrices with real or complex behave with respect to eigenvalue questions in the same way as nonhermitian Gaussian random matrices. For example, their eigenvalue distribution converges always to a circular law. Are they the $N\times N$ random matrices with real or complex entries asymptotically free?

$\begingroup$ asymptotic freedom means that the large$N$ eigenvalue distribution of the sum $A+UBU^\ast$, with $U$ an arbitrary unitary matrix, depends only the large$N$ eigenvalue distributions of the matrices $A$ and $B$. This is not possible if the ensemble lacks unitary invariance. $\endgroup$– Carlo BeenakkerJul 1 '18 at 14:24
If $A_N$ and $B_N$ are two independent Girko ensembles then one should have that they are asymptotically $*$free; the same should be true if the second ensemble $B_N$ is a sequence of deterministic matrices with a limiting $*$distribution.
I am at the moment not aware that this is written down in this form somewhere, but the proofs should be similar to the proofs that independent Wigner matrices and deterministic matrices are asymptotically free.