I assume the matrices $X$ and $Y$ are independent. Since the trace commutes with the expectation value, and since the expectation value of the product of independent random variables is the product of expectation values, we have $$ F(a)=\mathbb{E}\bigl[{\rm tr}\,\bigl({(aX{X^H} + I)^{ - 1}}Y{Y^H}\bigr)\bigr]={\rm tr}\bigl(\,\mathbb{E}[(aX{X^H} + I)^{ - 1}]\mathbb{E}[Y{Y^H}]\bigr).$$ The second factor is simply $K$ times the unit matrix, so $$F(a)=K\,{\rm tr}\,\mathbb{E}[(aX{X^H} + I)^{ - 1}].$$ We can now again exchange trace and expectation value, to rewrite this as an integral over the eigenvalues $\mu_k$ of $XX^H$, with density $\rho(\mu)$, $$F(a)=K\int \rho(\mu)(a\mu+1)^{-1}\,d\mu.$$ The density $\rho(\mu)$ is known, for large matrix size it is the Marcenko-Pastur distribution.

I assume the matrices $X$ and $Y$ are independent. Since the trace commutes with the expectation value, and since the expectation value of the product of independent random variables is the product of expectation values, we have $$ F(a)=\mathbb{E}\bigl[{\rm tr}\,\bigl({(aX{X^H} + I)^{ - 1}}Y{Y^H}\bigr)\bigr]={\rm tr}\bigl(\,\mathbb{E}[(aX{X^H} + I)^{ - 1}]\mathbb{E}[Y{Y^H}]\bigr).$$ The second factor is simply $K$ times the unit matrix, so $$F(a)=K\,{\rm tr}\,\mathbb{E}[(aX{X^H} + I)^{ - 1}].$$ We can now again exchange trace and expectation value, to rewrite this as an integral over the eigenvalues $\mu_k$ of $XX^H$, with density $\rho(\mu)$, $$F(a)=K\int \rho(\mu)(a\mu+1)^{-1}\,d\mu.$$ The density $\rho(\mu)$ is known, for large matrix size it is the Marcenko-Pastur distribution.

_{For the Marcenko-Pastur distribution, so for $M\geq N\gg 1$, I find $$F(a)=\frac{K}{2a} \left(\sqrt{a^2 (M-N)^2+2 a (M+N)+1}+a (N-M)-1\right).$$}