Actually, since you know that $A(t)$ is diagonal, you can easily write $A(t) = -a(t)^{-1}a'(t)$ where $a(t)$ is a diagonal matrix satisfying $a(0)=I$ (just integrate and exponentiate each diagonal term separately). Hence you can regard $a(t)$ as known and write the equation in the form $$ y'(t) = - a(t)^{-1}a'(t)y(t) + B(t)\,\overline{y(t)}, $$ which gives $$ (a(t)y(t))' = a(t)B(t)\overline{(a(t)^{-1})}\,\,\overline{a(t)y(t)}. $$ Set $z(t) = a(t)y(t)$ and write $L(t) = a(t)B(t)\overline{(a(t)^{-1})} C$, where $C:\mathbb{C}^n\to\mathbb{C}^n$ is the $\mathbb{R}$-linear map of conjugation. Then the equation becomes the $\mathbb{R}$-linear system of ODE $$ z'(t) = L(t) z(t). $$ Of course, $L(t)$, which can be regarded as known, can be written in the form $L(t) = -b(t)^{-1}b'(t)$ where $b(t)$ takes values in the matrix Lie group $\mathrm{Aut}_\mathbb{R}(\mathbb{C}^n,\mathbb{C}^n)$, and satisfies $b(0)=I$. The general solution of this equation is then $$ z(t) = b(t)^{-1}z(0) \qquad \text{so}\qquad y(t) = a(t)^{-1}b(t)^{-1}y(0). $$ Now, the Liouville formula yields $$ \det(a(t)) = \exp\left(-\int_0^t \mathrm{tr}(A(\tau))\,d\tau\right) $$ $$ \det(b(t)) = \exp\left(-\int_0^t \mathrm{tr}(L(\tau))\,d\tau\right). $$ (Here, you must remember to compute the trace of $L(\tau)$ as an $\mathbb{R}$-linear transformation of $\mathbb{C}^n$, and the determinant of $b$ as an $\mathbb{R}$-linear transformation of $\mathbb{C}^n$.) Thus, $y(t) = \Phi(t)y(0)$ where, suitably interpreted, we have $$ \det(\Phi(t)) = \exp\left(\int_0^t \mathrm{tr}\bigl(A(\tau)+L(\tau)\bigr)\,d\tau\right) $$ In other words, this determinant can be found by quadrature, just as in the linear case.