$\newcommand\ep\varepsilon$This is impossible to do even for $d=1$, as your [model is not identifiable][1], that is, the parameters of the model are not identifiable even if the distribution of the $X_i$ is fully known. 

Indeed, let $R_1,\dots,R_n$'s be independent Rademacher random variables, with $P(R_i=\pm1)=1/2$. Then the following two scenarios result in the same distribution of the $X_i$'s: 

(i) $\mu=0$ and $\ep_i=R_i$ for all $i$;

(ii) $\mu=1$, and $O_i=R_i$ and $\ep_i=0$ for all $i$.  

We see that even the function $1(\mu=0)$ of the parameter $\mu$ is not identifiable, that is, the value of 
$1(\mu=0)$ cannot be determined even if the distribution of the $X_i$ is fully known.


  [1]: https://en.wikipedia.org/wiki/Identifiability#Definition