$\newcommand\ep\varepsilon$This is impossible to do even for $d=1$, as your model is not identifiable, that is, the parameters of the model are not identifiable even if the distribution of the $X_i$'s is fully known.
Indeed, let $R_1,\dots,R_n$'s be independent Rademacher random variables, with $P(R_i=\pm1)=1/2$. Then the following two scenarios result in the same distribution of the $X_i$'s:
(i) $\mu=0$ and $\ep_i=R_i$ for all $i$;
(ii) $\mu=1$, and $O_i=R_i$ and $\ep_i=0$ for all $i$.
We see that even the function $1(\mu=0)$ of the parameter $\mu$ is not identifiable, that is, the value of $1(\mu=0)$ cannot be determined even if the distribution of the $X_i$'s is fully known.