Take $X=\mathbb C$, $U=\mathbb C^\times$, and $\mathcal L$ a nontrivial rank 1 local system (with monodromy $\mu \neq 1$, say).
Then the singular support of $IC(U,\mathcal L)$ is the union $T^\ast _X X \cup T^\ast_0 X$ of the zero section (which is what your conjecture would give in this case) with the cotangent fiber.
Note that here we have $ j_\ast \mathcal L \cong IC(U,\mathcal L) \cong j_! \mathcal L$$ j_\ast \mathcal L[1] \cong IC(U,\mathcal L) \cong j_! \mathcal L[1]$. One can compute the singular support either using the sheaf definition from Kashiwwara-Schapira, or by considering the associated $D$-module $D_{\mathbb C}/D_{\mathbb C}(x\partial_x - \log(\mu))$ under the Riemann-Hilbert correspondence.