Your conjecture is correct.
Suppose that we have a compact unit neighbourhood $U$ such that $\mu(UxU) \ll 1$ for all $x$. As you have already noted, we can take $\mu$ to be unimodular, and the choice of neighbourhood is not relevant, so we may assume without loss of generality that $U$ is symmetric: $U^{-1} = U$. (This makes $U$ what is called an approximate group in arithmetic combinatorics, and the intuition that $U$ should behave like a subgroup of $G$ is underlying the arguments below.) We allow implied constants in asymptotic notation to depend on $U$, thus for instance $\mu(U) \asymp 1$.
Note that the conjugate $x U x^{-1}$ of $U$ is commensurate with $U$ in the sense that $\mu( U \cdot x U x^{-1} ) = \mu( U x U ) \ll 1$. (In the language of arithmetic combinatorics, $U$ stays close to its conjugates $xUx^{-1}$ in Ruzsa distance.) In the spirit of the isomorphism theorems (or the Ruzsa covering lemma in arithmetic combinatorics), one now expects $U$ and $xUx^{-1}$ to have large intersection, and this can be accomplished (at the cost of enlarging $U$ to $U^2$) by the following convolution argument (cf. the double counting argument that shows that $|H \cdot K| = |H| |K| / |H \cap K|$ for finite subgroups $H,K$ of $G$). Observe that the convolution $1_U * 1_{xUx^{-1}}$ has an $L^1(G,\mu)$ norm of $\mu(U) \mu(x U x^{-1}) \asymp 1$ and is supported on $U x U x^{-1}$, which has measure $O(1)$. Thus there must exist a point $y \in G$ where $1_U * 1_{xUx^{-1}}(y) \gg 1$, thus
$$ \mu( yU \cap x U x^{-1} ) \gg 1$$
which implies
$$ \mu( (yU \cap x U x^{-1})^{-1} \cdot (yU \cap x U x^{-1}) ) \gg 1$$
and hence
$$ \mu( U^2 \cap x U^2 x^{-1} ) \gg 1.$$
To put it another way, the inner products of the functions $1_{x U^2 x^{-1}}$ with $1_{U^2}$ are uniformly bounded from below.
We can now use an "ergodic" argument to extract an invariant object (in the spirit of the Alaoglu--Birkhoff ergodic theorem).
Let $S$ be the closed convex hull in $L^2(G,\mu)$ of the functions $1_{x U^2 x^{-1}}$. By the Hilbert projection theorem, this set has a unique element $f$ of minimal norm. All elements of $S$ have inner product with $1_{U^2}$ uniformly bounded from below, so $f$ does also; in particular, $f$ is non-trivial. Since $S$ is conjugation-invariant, symmetric, bounded and consists of non-negative functions, $f$ must be non-negative, symmetric, and conjugation-invariant.
At this point one could already extract a conjugation-invariant set of positive finite measure by taking level sets of $f$, but this is not quite regular enough for the conjecture, so we take a convolution to achieve an additional smoothing (in the spirit of the Steinhaus theorem). The convolution $f*f$ is then in $C_0(G)$ (this follows from a standard limiting argument, approximating $f$ in $L^2(G,\mu)$ by $C_c(G)$ functions and using Young's inequality), conjugation-invariant, and strictly positive at the origin; taking level sets, we obtain a non-trivial conjugation-invariant compact unit neighbourhood, as required.
