This is not true, even for the case where $A$ is a quantum state. Notation-wise, I will eschew exponentials of $\text{ad}$ and instead write the unitary transformations explicitly. I will also use the notation $\rho$ for a quantum state, instead of $A$.

There is a nice counterexample from physics known as the depolarizing channel. Consider a single-qubit quantum state $\rho$. Apply one of the four Pauli matrices $I$, $\sigma^x$, $\sigma^y$, $\sigma^z$ with equal probability. Note that these are all Hermitian and unitary matrices.

Then $\rho \to \frac{1}{4}(\rho + \sigma^x \rho \sigma^x + \sigma^y \rho \sigma^y + \sigma^z \rho \sigma^z)$

However, as you can check, the output commutes with the Pauli matrices and so must be proportional to the identity (just Schur's lemma).

That is, $\rho \to \frac{1}{2} I$ for all normalized quantum states $\rho$.

In particular, this is not an invertible transformation, since all quantum states $\rho$ are mapped to the identity.

This is not true, even for the case where $A$ is a quantum state. Notation-wise, I will eschew exponentials of $\text{ad}$ and instead write the unitary transformations explicitly. I will also use the notation $\rho$ for a quantum state, instead of $A$.

There is a nice counterexample from physics known as the depolarizing channel. Consider a single-qubit quantum state $\rho$. Apply one of the four Pauli matrices $I$, $\sigma^x$, $\sigma^y$, $\sigma^z$ with equal probability. Note that these are all Hermitian and unitary matrices.

Then $\rho \to \frac{1}{4}(\rho + \sigma^x \rho \sigma^x + \sigma^y \rho \sigma^y + \sigma^z \rho \sigma^z)$

However, as you can check, the output commutes with the Pauli matrices and so must be proportional to the identity (just Schur's lemma).

That is, $\rho \to \frac{1}{2} I$ for all normalized quantum states $\rho$. (More generally, this transformation would send a generic single-qubit operator $A$ to $\frac{\text{Tr}[A]}{2} I$.)

In particular, this is not an invertible transformation, since all quantum states $\rho$ are mapped to the identity.

I have an additional comment. It's clear that the transformation above is not invertible, but the transformation cannot send any quantum states to $0$! By taking the trace of some generic $E(A)$ made by probabilistic linear combinations of conjugating by unitaries, it's easy to see that the trace of the matrix $A$ is preserved, so if $A$ is not traceless, $E(A)$ cannot be the zero matrix.

Please note that the set of operators corresponding to quantum states is only a subset and not a linear subspace of the set of operators. I believe you're making a mistake when identifying lack of a kernel on this subset of operators with invertibility on this subset of operators. From the depolarizing channel example above, it's clear that one can have $E(\rho) \neq 0$ for all quantum states $\rho$, even when $E$ is not invertible.