Yes, this can happen.
Let's start in $L$ and first perform Easton forcing to add a Cohen
subset to every regular cardinal, giving rise to the forcing
extension $L[G]$. Next, over $L[G]$ we force to code every set into
the GCH pattern, for example, by iterating the forcing that forces
GCH or its negation with a lottery sum at each such cardinal stage.
In the resulting forcing extension $V=L[G][H]$, we therefore have
$\newcommand\HOD{\text{HOD}}\HOD^V=V$, since every set is coded.
Next, over $V$, force to add a $V$-generic class
$A$ of the regular cardinals. This forcing adds no sets,
and $A$ is not a definable class in $V$, but we may consider it in
GBC. Let $M=L[G\upharpoonright A]$ be the inner model arising by
taking only the sets of $G$ at coordinates in $A$. So this is a
transitive inner model of $V$, but it is not first-order definable
in $V$, since $A$ is generic over $V$.
But meanwhile,
$\text{HOD}(M)$ as computed in $V$ will be the same as $\HOD$,
since it is already all of $V$ and so the extra parameters of $M$
didn't add anything.
So we have an inner model $M$ that is not definable in $\HOD(M)$,
as desired.
Let's now modify the example to produce a model $V$ of ZFC with a
definable inner model $M$, such that $M$ is not definable in
$\HOD(M)$. Start in $L$ as before, and this time, add the class $A$
first, over $L$. In particular, every initial segment of $A$ is in
$L$. Now, over the GBC model $L[A]$, force to add Cohen sets at
every uncountable regular cardinal, forming $L[A][G]$. Finally,
using product forcing rather than an iteration, we may force to
code every set in $L[G]$, and also the classes $G$ and $A$, into
the GCH pattern, producing an extension $V=L[A][G][H]$ in which
$\HOD=L[G]$. (For example, the methods of my paper G. Fuchs, J. D.
Hamkins, J. Reitz, Set-theoretic
geology show how to
do this, while also controlling the mantle and the generic $\HOD$.)
Let $M=L[G\upharpoonright A]$. This is a transitive inner model of
ZFC. It is definable in $V$, since the class $G\upharpoonright A$
is definable there. Further, since every initial segment of $A$ is
in $L$, it follows that $L[G\upharpoonright A]\subset L[G]$, which
is the $\HOD$ of $V$, and so $\HOD(M)=\HOD=L[G]$ in $V$. But $A$ is
not definable in $L[G]$, since $A$ and $G$ are mutually generic. So
$M$ is not definable in $L[G]$.
Which is to say, $M$ is a definable transitive inner model of $V$,
but it is not definable in $\HOD(M)$, as you desired.
