I think this case is actually not so obvious. The issue is that to derive $R\mathscr Hom$ in the first variable you would need to use a locally free resolution while $Rf_*$ being a covariant right derived functor would need an injective one. If you look at the usual proofs of the various forms of GSS, you'll see that the idea is to take a resolution of the first object in the first category, apply the first (derived) functor and then see that what you get in the second category computes the second derived functor. With this, as it is, you would need to switch resolutions and I am not sure I would want to get into that computation.

On the other hand, if your situation is such that $R^if_*\mathscr O_X=0$ for $i>d$ for some $d\in \mathbb Z$, for instance, $f$ is proper, $Y$ is (locally) Noetherian and $f$ has fibers of dimension at most $d$, then you can take your starting functor $R^df_*$ and consider $Rf_*\mathscr O_X$ as the *left* derived functor of that. (See Prop 7.4 on page 74 in [Hartshorne, R&D].) I would expect that if you know that

- $R^if_*\mathscr F=0$ for $i>d$ for every coherent $\mathscr F$, and
- $R^df_*\neq 0$,

then this should work. For simplicity let me assume that $f$ is a projective morphism of schemes of finite type over a field with $d$ equal the maximal fiber dimension of $f$. If you need it in more generality you can probably adjust the argument. (If it works!!).

Namely, fix a coherent sheaf $\mathscr F$ on $X$ and take a locally free resolution $\mathscr L^\bullet$ of $\mathscr F$. (**Addendum**: this should probably consist of locally free sheaves of the form $\oplus\mathscr O(-n)$ as we discussed in the comments, so the $R^df_*$ of this sequence would indeed be a locally free resolution of $R^df_*\mathscr F$ as claimed in the next sentence.) Then by cohomology and base change $R^df_*\mathscr L^i$ is locally free for all $i$ and hence $R^df_*\mathscr L^\bullet$ is a complex of locally free sheaves which is quasi-isomorphic to $\left(L(R^df_*\mathscr)\right)(F)$ (=the (total) left derived functor of $R^df_*$ applied to $\mathscr F$). Now if you just follow the usual proof, you should (check this!!!) get a spectral sequence something like
$$
E_2^{p,q}= \mathscr Ext^p(\underbrace{L^q(R^df_*)}_{R^{d-q}f_*}\mathscr F, \mathscr G)\Rightarrow
E_\infty^{p,q}= h^{p+q}\left( R\mathscr Hom(Rf_*\mathscr F, \mathscr G)\right)
$$

Actually I think that to do this right, here probably both $p$ and $q$ should be negative, since the locally free resolution goes that way. A simple first check is to see if you can write down the edge maps from obvious short exact sequences. For $E_2$ spectral sequences, the maps are usually compositions of two edge maps from two different short exact sequences.

So, in terms of the original functors this is not a first quadrant spectral sequence, but one should expect that anyway.

Anyway, as it is probably clear from what I have written, I did not work out the details, so it is quite possible that there is some issue I didn't consider and also, clearly I didn't even give you an explicit answer, but perhaps you can work it out. And, please do work out the details carefully (and perhaps post it here if you have).