There doesn't exist a mathematical publication by Grothendieck explicitly presenting that formalism. It was going to be addressed in "Exposé 0" of SGA 5, but the editors excluded it from the final publication*. Of course, it is implicit in each of Grothendieck's duality theories:
- For coherent sheaves (Serre duality), called "continuous coefficients", the reference at that time was Residues and Duality, which were the notes of a seminar given by Hartshorne, based on Grothendieck's draft. This was the intended subject of what was going to be EGA X.
- For étale cohomology (Poincaré duality), described as "discrete coefficients", of course, SGA 4 and 5 are the standard source.
- The formalism is also behind Verdier's treatment of Poincaré duality in locally compact spaces and analytic spaces. These works showed the validity of the formalism outside algebraic geometry.
*Grothendieck talks about it extensively in Récoltes et Semailles (I don't know much about the other side of the story).
EDIT: Gallimard's version of ReS includes additional notes by Grothendieck, written in 1986, and not present in the one I was consulting. In one of these notes, he says that, according to Illusie, this exposé was, in fact, was written with the name "Applications de la dualité globale et formules de Künneth", and Grothendieck gave permission to "dismember it" and partition its contents between the following exposés:
- SGA 4 exposés XVII and XVIII.
- SGA 5 exposés I and II. Exposé II was, in the end, moved to Deligne's "SGA 4 1/2".
Grothendieck thinks of this as a mistake, and doesn't believe it's true.
About the destiny of that Exposé 0 (and of a previous, introductory Exposé which was also forgotten), he writes (old version of ReS) :
Je viens à l’instant de parcourir mes notes manuscrites pour les premiers trois exposés de SGA 5, notes qu’Illusie a bien voulu me retourner l’an dernier à ma demande. (Il est le seul des ex-rédacteurs qui ait pris la peine de me restituer les notes que je leur avais confiées. . .) Le premier exposé (i.e. the introductory Exposé) consistait en un vaste "tour d’horizon" de ce qui avait été accompli dans le séminaire précédent SGA 4, en ce qui concerne le formalisme cohomologique étale et ses relations à divers autres contextes. Le deuxième exposé (this is the Exposé 0 which I referred to) développe en long et en large le formalisme "abstrait" des six variances. Il y a un formulaire essentiellement complet, mais sans effort encore pour cerner les compatibilités entre isomorphismes canoniques. (C’était là une tâche de nature plus technique, inutile à un moment où je tenais avant tout à "faire passer" ce yoga de dualité, dont je sentais bien tout la force.) Inutile de dire qu’il n’y a trace dans l’édition-Illusie ni de l’un ni de l’autre exposé.
I think Grothendieck wished to express the formalism in a general framework of derived and triangulated categories, just like he did with his Riemann-Roch theorem in SGA 6.
The theory of derived and triangulated categories was developed by Verdier in his thesis, but he decided against publishing it. Nevertheless, an embryonic version appeared as an État 0, from where (I think) they took the part about derived categories in SGA 4 1/2. The full work only saw the light of day in 1996 (posthumously), thanks to Illusie's efforts.
You can also find here a handwritten "cheatsheet" by Grothendieck, summarizing the "formalisme des six opérations".