Grothendieck spectral sequence when one of the functors is contravariant

Question

Let $f \colon X \rightarrow S$ be a morphism of schemes. I am interested in computing the cohomology groups of $$ \mathbf{R}\mathscr{H}om(\mathbf{R}f_* \mathcal{O}_X, \mathcal{O}_S) $$ in terms of $\mathrm{Ext}^p(R^qf_* \mathcal{O}_X, \mathcal{O}_S)$. Is there a spectral sequence that achieves this? If so, what are the entries of its pages, how do the differentials go, which way does the eventual filtration on the infinity page go, etc.? Any references to the literature where this is explained would also be highly appreciated.

I am familiar with the Grothendieck spectral sequence for a composition of functors, but I cannot see how that applies because $\mathbf{R}f_*$ is covariant and $\mathbf{R}\mathscr{H}om$ is contravariant (in the first variable).

As alluded to in the title, the question may be posed in much greater generality, but let me stick with the above for the sake of concreteness.

EDIT. I realized that I forgot that the usual Grothendieck spectral sequence has a condition that the first functor maps injectives to acyclics for the second functor. A condition of this sort does not seem to hold in my setting, so perhaps there is no spectral sequence of the type I want. Further comments are nevertheless appreciated!

Answer 1

I think this case is actually not so obvious. The issue is that to derive $R\mathscr Hom$ in the first variable you would need to use a locally free resolution while $Rf_*$ being a covariant right derived functor would need an injective one. If you look at the usual proofs of the various forms of GSS, you'll see that the idea is to take a resolution of the first object in the first category, apply the first (derived) functor and then see that what you get in the second category computes the second derived functor. With this, as it is, you would need to switch resolutions and I am not sure I would want to get into that computation.

On the other hand, if your situation is such that $R^if_*\mathscr O_X=0$ for $i>d$ for some $d\in \mathbb Z$, for instance, $f$ is proper, $Y$ is (locally) Noetherian and $f$ has fibers of dimension at most $d$, then you can take your starting functor $R^df_*$ and consider $Rf_*\mathscr O_X$ as the left derived functor of that. (See Prop 7.4 on page 74 in [Hartshorne, R&D].) I would expect that if you know that

1. $R^if_*\mathscr F=0$ for $i>d$ for every coherent $\mathscr F$, and
2. $R^df_*\neq 0$,

then this should work. For simplicity let me assume that $f$ is a projective morphism of schemes of finite type over a field with $d$ equal the maximal fiber dimension of $f$. If you need it in more generality you can probably adjust the argument. (If it works!!).

Namely, fix a coherent sheaf $\mathscr F$ on $X$ and take a locally free resolution $\mathscr L^\bullet$ of $\mathscr F$. (Addendum: this should probably consist of locally free sheaves of the form $\oplus\mathscr O(-n)$ as we discussed in the comments, so the $R^df_*$ of this sequence would indeed be a locally free resolution of $R^df_*\mathscr F$ as claimed in the next sentence.) Then by cohomology and base change $R^df_*\mathscr L^i$ is locally free for all $i$ and hence $R^df_*\mathscr L^\bullet$ is a complex of locally free sheaves which is quasi-isomorphic to $\left(L(R^df_*\mathscr)\right)(F)$ (=the (total) left derived functor of $R^df_*$ applied to $\mathscr F$). Now if you just follow the usual proof, you should (check this!!!) get a spectral sequence something like $$ E_2^{p,q}= \mathscr Ext^p(\underbrace{L^q(R^df_*)}_{R^{d-q}f_*}\mathscr F, \mathscr G)\Rightarrow E_\infty^{p,q}= h^{p+q}\left( R\mathscr Hom(Rf_*\mathscr F, \mathscr G)\right) $$

Actually I think that to do this right, here probably both $p$ and $q$ should be negative, since the locally free resolution goes that way. A simple first check is to see if you can write down the edge maps from obvious short exact sequences. For $E_2$ spectral sequences, the maps are usually compositions of two edge maps from two different short exact sequences.

So, in terms of the original functors this is not a first quadrant spectral sequence, but one should expect that anyway.

Anyway, as it is probably clear from what I have written, I did not work out the details, so it is quite possible that there is some issue I didn't consider and also, clearly I didn't even give you an explicit answer, but perhaps you can work it out. And, please do work out the details carefully (and perhaps post it here if you have).

Answer 2

If $I^*$ and $J^*$ are global sections of appropriate injective resolutions of $f_*{\cal O}_X$ and ${\cal O}_S$, then you have a double complex that looks in part like this:

$$\matrix{ Hom(I^q,J^p)&\rightarrow&Hom(I^{q},J^{p+1})\cr &&\uparrow\cr &&Hom(I^{q+1},J^{p+1})&\rightarrow&Hom(I^{q+1},J^{p+2})}$$

Your $E_2$ terms come from taking cohomology vertically and then horizontally, so your $E_2$ differentials should go from $E_2^{p,q}$ to $E_2^{p+2,q+1}$.