This is not an answer, but a collection of observations and ideas, in hopes that it helps someone to get more.
Let us ask a bit more of our function:
- If $AB$ is positive semidefinite, then $\Phi(AB) = \Phi(A)\Phi(B)$.
- $\Phi(H)$ is nonsingular for at least one $H \in \mathcal{H}$. (I think this is quite a reasonable request to make)
Notice that the first of these two requests also means that if $A,B$ commute, then $\Phi(A),\Phi(B)$ commute as well. Using this, we easily see that
$$t\Phi(H) = \Phi(t{\rm I}H) = \Phi(t{\rm I})\Phi(H), \quad \text{for all $H$}.$$
We pick $H$ such that $\Phi(H)$ is nonsingular and get
$$\Phi(t{\rm I}) = t{\rm I}, \quad \text{for all $t \ge 0$}.$$
This also draws a trivial conclusion that
$$\Phi(H^{-1}) = \Phi(H)^{-1}.$$
If $A$ is a matrix whose elements are all equal to $1$, then
$$nf(A) = f(nA) = f(A^2) = f(A)^2.$$
Unfortunately, all-ones matrices are not the only ones with such property, so I cannot get anything more from this (I was hoping to get some insight on $\Phi(A)$, at least for such special $A$).
The preservation of commutativity gave me the following idea. If $A,B \in \mathcal{H}$ commute, then there exist a unitary matrix $U$ and nonnegative diagonal matrices $\Lambda_A,\Lambda_B$ such that
$$A = U \Lambda_A U^*, \quad B = U \Lambda_B U^*.$$
But, since this means that $\Phi(A),\Phi(B)$ also commute, there exist a unitary matrix $V$ and upper triangular matrices $T_A,T_B$ such that
$$\Phi(A) = V T_A V^*, \quad \Phi(B) = V T_B V^*.$$
So, it may make sense to define $\Phi(X)$ as a map $(U,\Lambda) \mapsto (V, T)$, where $X = U \Lambda U^*$ for $U$ unitary and $\Lambda$ nonnegative diagonal, and $\Phi(X) = V T V^*$ for $V$ unitary and $T$ uppertriangular.
The problems with this approach are:
- defining such map to guarantee that $\Phi(X)$ is nonnegative,
- property 2 gets hard to prove for noncommutative matrices.
The first problem might be solved by changing the requirements on $V$. Maybe it can be some nonsingular matrix with nonnegative elements (so, not unitary). The question of how to get such matrix remains.
I'm currently at loss for additional ideas. I hope these will help at least a bit.