First, some general comments.
If $f:X\to S$ is an $S$-scheme, put $A(X):=f_*(\mathscr{O}_X)$. If $X$ and $Y$ are $S$-schemes, we have a natural map $$\mathrm{Hom}_S(X,Y)\to\mathrm{Hom}_{\mathscr{O}_S-\text{algebras}}(A(Y), A(X))$$ which is bijective if $Y$ is $S$-affine (EGA I, (9.1.5); this is true even if $X$ is an $S$-locally ringed space).
It follows formally that $X$ has an $S$-affine hull if and only if $A(X)$ has a "quasicoherent co-hull" in the category of $\mathscr{O}_S$-algebras (a trivial case where this holds is when $A(X)$ happens to be quasicoherent).
We may ask more generally
which $\mathscr{O}_S$-modules $M$ have a quasicoherent co-hull
i.e. are such that the functor $F\mapsto\mathrm{Hom}_{\mathscr{O}_S-\text{modules}}(F,M)$ on quasi-coherent $\mathscr{O}_S$-modules is representable.
For instance, this holds if $M$ is a submodule of a quasicoherent $G$: in this case, the above functor is represented by the largest quasicoherent submodule of $M$, which exists by Lemma Tag 01QZ on the Stacks Project, cited by Matthieu.
We can use this fact for general $M$: if $u:F\to M$ is a morphism (with $F$ quasicoherent), let $K(u)$ be the largest quasicoherent submodule of $\mathrm{Ker}(u)$. Then $u$ factors through $I(u):=F/K(u)$, which is quasicoherent, and the induced $\overline{u}:I(u)\to M$ has the property that there is no nontrivial quasicoherent submodule of $I(u)$ killed by $\overline{u}$. Let us call this property "qc-injectivity".
For given $M$, denote by $S(M)$ the category of all qc-injective $F\to M$ (with $F$ quasicoherent, of course). The following facts are easy to check:
- $S(M)$ is essentially an ordered set (i.e. Hom sets have at most one element).
- Every map in $S(M)$ is injective as a map of sheaves.
- $S(M)$ has arbitrary colimits.
- If $M$ has a quasicoherent co-hull $M_0$, then $S(M)$ is (equivalent to) the ordered set of quasicoherent subsheaves of $M_0$.
This suggests to define the quasicoherent co-hull of $M$ as the colimit of the obvious functor from $S(M)$ to quasicoherent modules. Formally, this works, except that $S(M)$ may be too big, and the question becomes:
When is the category $S(M)$ essentially small?
Remarks:
- If $M$ is an $\mathscr{O}_S$-algebra, a quasicoherent co-hull of $M$ as an $\mathscr{O}_S$-module is automatically an $\mathscr{O}_S$-algebra (and a co-hull in that category), and conversely a quasicoherent co-hull of $M$ in the category of $\mathscr{O}_S$-algebras will be a quasicoherent co-hull in the category of $\mathscr{O}_S$-modules. (This follows from the fact that if $F$ is quasicoherent, so is its symmetric algebra).
- If $S$ is affine, any $M$ has a quasicoherent co-hull, which is $\widetilde{\Gamma(S,M)}$.
Edited on Sep. 6, 2014: Here are some partial results, not using the "small category" criterion stated above. The main result is:
Theorem 1. Assume $S$ is quasicompact and quasiseparated. Then every $\mathscr{O}_S$-module has a quasicoherent co-hull. Consequently, every $S$-scheme has an affine hull.
From now on, we denote by $\mathrm{qcch}(S,M)$ or $\mathrm{qcch}(M)$ the quasicoherent co-hull of the $\mathscr{O}_S$-module $M$, when it exists. Obviously, $\mathrm{qcch}$ commutes with finite products of sheaves.
Recall that $\mathrm{qcch}(M)$ exists when $M$ embeds into a quasicoherent module, or when $S$ is affine. More generally:
Lemma 1. If $\mathrm{qcch}(M)\xrightarrow{j}M$ exists and $N$ is a submodule of $M$, then $\mathrm{qcch}(N)$ exists, and in fact is $\mathrm{qcch}(j^{-1}(N))$.
(The point is that $\mathrm{qcch}(j^{-1}(N))$ exists because $j^{-1}(N)$ is a subsheaf of $\mathrm{qcch}(M)$ which is quasicoherent. Checking the universal property is then obvious. Alternatively, using the above criterion, observe that $S(N)$ is a subcategory of $S(M)$.)
Lemma 2. Let $f:X\to S$ be a morphism, and let $M$ be an $\mathscr{O}_X$-module. Assume $F:=\mathrm{qcch}(X,M)$ exists. Then we have the equality
$$\mathrm{qcch}(S,f_*M)=\mathrm{qcch}(S,f_*F)$$
(with the convention that if one side exists, so does the other).
In particular, if $f$ is quasicompact and quasiseparated, then $\mathrm{qcch}(S,f_*M)$ exists if $\mathrm{qcch}(X,M)$ does, and in this case $\mathrm{qcch}$ commutes with $f_*$.
(Straightforward from the fact that $f^*$ preserves quasicoherence.)
Lemma 3. Let $u_i:S_i\hookrightarrow S$ ($i=1,\dots,n$) be a finite sequence of quasicompact open immersions that cover $S$. Let $M$ be an $\mathscr{O}_X$-module. Assume $\mathrm{qcch}(S_i,u_i^*M)$ exists for each $i$. Then $\mathrm{qcch}(S,M)$ exists.
Proof: put $M_i:=u_{i*}u_i^*M$. By Lemma 2, each $\mathrm{qcch}(S,M_i)$ exists. But $M\subset\prod_{i=1}^n M_i$, so the result follows from Lemma 1.
Proof of Theorem 1: follows from the affine case together with Lemma 3, by taking for $(S_i)$ any finite affine covering of $S$.
Remark. Assume $S$ is quasiseparated, and let $(S_i)_{i\in I}$ be an affine open covering. Since the immersions $S_i\hookrightarrow S$ are quasicompact, by repeating the arguments of Lemma 2 and Lemma 3 we see that the existence of $\mathrm{qcch}(M)$ for arbitrary $M$ reduces to the case $M=\prod_{i\in I}F_i$ where each $F_i$ is quasicoherent.
