Actually, since you know that $A(t)$ is diagonal, you can easily write $A(t) = -a(t)^{-1}a'(t)$ where $a(t)$ is a diagonal matrix satisfying $a(0)=I$ (just integrate and exponentiate each diagonal term separately).

Hence you can regard $a(t)$ as known and write the equation in the form $$ y'(t) = - a(t)^{-1}a'(t)y(t) + B(t)\,\overline{y(t)}, $$ which gives $$ (a(t)y(t))' = a(t)B(t)\overline{(a(t)^{-1})}\,\,\overline{a(t)y(t)}. $$ Set $z(t) = a(t)y(t)$ and write $L(t) = a(t)B(t)\overline{(a(t)^{-1})} C$, where $C:\mathbb{C}^n\to\mathbb{C}^n$ is the $\mathbb{R}$-linear map of conjugation. Then the equation becomes the $\mathbb{R}$-linear system of ODE $$ z'(t) = L(t) z(t). $$ Of course, $L(t)$, which can be regarded as known, can be written in the form $L(t) = -b(t)^{-1}b'(t)$ where $b(t)$ takes values in the matrix Lie group $\mathrm{Aut}_\mathbb{R}(\mathbb{C}^n,\mathbb{C}^n)$, and satisfies $b(0)=I$. The general solution of this equation is then $$ z(t) = b(t)^{-1}z(0) \qquad \text{so}\qquad y(t) = a(t)^{-1}b(t)^{-1}y(0). $$

Now, the Liouville formula yields $$ \det(a(t)) = \exp\left(-\int_0^t \mathrm{tr}(A(\tau))\,d\tau\right) $$ $$ \det(b(t)) = \exp\left(-\int_0^t \mathrm{tr}(L(\tau))\,d\tau\right). $$ (Here, you must remember to compute the trace of $L(\tau)$ as an $\mathbb{R}$-linear transformation of $\mathbb{C}^n$, and the determinant of $b$ as an $\mathbb{R}$-linear transformation of $\mathbb{C}^n$.)

Thus, $y(t) = \Phi(t)y(0)$ where, suitably interpreted, we have $$ \det(\Phi(t)) = \exp\left(\int_0^t \mathrm{tr}\bigl(A(\tau)+L(\tau)\bigr)\,d\tau\right) $$ In other words, this determinant can be found by quadrature, just as in the linear case.

NB: I'll slightly rearrange this for clarity:

As I should have remarked at the beginning, writing the fundamental solution to your system in the form $$ y(t) = \Phi(t)\,y(0), $$ where $\Phi(0)=I$, is only possible when you allow $\Phi(t)$ to take values in $$\mathrm{Aut}_\mathbb{R}(\mathbb{C}^n,\mathbb{C}^n) \simeq \mathrm{GL}(2n,\mathbb{R}), $$ i.e., you can't assume that $\Phi(t):\mathbb{C}^n\to\mathbb{C}^n$ is $\mathbb{C}$-linear, but only $\mathbb{R}$-linear. Consequently, the very meaning of 'determinant' is probably not what you had in mind, since the homomorphism $$ {\det}_\mathbb{R}:\mathrm{Aut}_\mathbb{R}(\mathbb{C}^n,\mathbb{C}^n)=\mathrm{GL}(2n,\mathbb{R})\to\mathbb{R}\setminus\{0\} $$ takes (nonzero) values in $\mathbb{R}$, and there is no meaningful way to `lift' this to a 'determinant' that assumes all (nonzero) values in the complex numbers. One should also note that, if $A:\mathbb{C}^n\to\mathbb{C}^n$ does happen to be $\mathbb{C}$-linear, then one has $$ {\det}_\mathbb{R}(A) = \bigl|{\det}_\mathbb{C}(A)\bigr|^2. $$

With that understood, there is a formula for $\det\bigl(\Phi(t)\bigr)$, it's just $$ \det(\Phi(t)) = \exp\left(\int_0^t \mathrm{tr}\bigl(A(\tau)+\overline{A(\tau)}\bigr)\,d\tau\right). $$

Here's how you can see this: Since you know that $A(t)$ is diagonal, you can easily write $A(t) = -a(t)^{-1}a'(t)$ where $a(t)$ is a diagonal matrix satisfying $a(0)=I$ (just integrate and exponentiate each diagonal term separately).

Hence you can regard $a(t)$ as known and write the equation in the form $$ y'(t) = - a(t)^{-1}a'(t)y(t) + B(t)\,\overline{y(t)}, $$ which gives $$ (a(t)y(t))' = a(t)B(t)\overline{(a(t)^{-1})}\,\,\overline{a(t)y(t)}. $$ Set $z(t) = a(t)y(t)$ and write $L(t) = a(t)B(t)\overline{(a(t)^{-1})} C$, where $C:\mathbb{C}^n\to\mathbb{C}^n$ is the $\mathbb{R}$-linear map of conjugation. Then the equation becomes the $\mathbb{R}$-linear system of ODE $$ z'(t) = L(t) z(t). $$ It's important to note that $\mathrm{tr}(L(t))$ vanishes identically.

Of course, $L(t)$, which can be regarded as known, can be written in the form $L(t) = -b(t)^{-1}b'(t)$ where $b(t)$ takes values in the matrix Lie group $\mathrm{Aut}_\mathbb{R}(\mathbb{C}^n,\mathbb{C}^n)$, and satisfies $b(0)=I$. The general solution of this equation is then $$ z(t) = b(t)^{-1}z(0) \qquad \text{so}\qquad y(t) = a(t)^{-1}b(t)^{-1}y(0). $$

Now, the Liouville formula yields $$ {\det}_\mathbb{C}(a(t)) = \exp\left(-\int_0^t \mathrm{tr}(A(\tau))\,d\tau\right) $$ $$ {\det}_\mathbb{R}(b(t)) = \exp\left(-\int_0^t \mathrm{tr}(L(\tau)+\overline{L(\tau)})\,d\tau\right). = 1 $$ (After all the trace of $L(t)$ as a real endomorphism is identically zero.)

Thus, $y(t) = \Phi(t)y(0)$ where $$ {\det}_\mathbb{R}(\Phi(t)) = \exp\left(\int_0^t \mathrm{tr}\bigl(A(\tau)+\overline{A(\tau)}\bigr)\,d\tau\right). $$