There doesn't exist a mathematical publication by Grothendieck explicitly presenting that formalism. It was going to be addressed in "Exposé 0" of SGA 5, but the editors excluded it from the final publication*. Of course, it is implicit in each of Grothendieck's duality theories:
- For coherent sheaves (Serre duality), called "continuous coefficients", the reference at that time was Residues and Duality, which were the notes of a seminar given by Hartshorne, based on Grothendieck's draft. This was the intended subject of what was going to be EGA X.
- For étale cohomology (Poincaré duality), described as "discrete coefficients", of course, SGA 4 and 5 are the standard source.
- The formalism is also behind Verdier's treatment of Poincaré duality in locally compact spaces and analytic spaces. These works showed the validity of the formalism outside algebraic geometry.
*Grothendieck talks about it extensively in Récoltes et Semailles (I don't know much about the other side of the story).
About the destiny of that Exposé 0 (and of a previous, introductory Exposé which was also forgotten), he writes:
Je viens à l’instant de parcourir mes notes manuscrites pour les premiers trois exposés de SGA 5, notes qu’Illusie a bien voulu me retourner l’an dernier à ma demande. (Il est le seul des ex-rédacteurs qui ait pris la peine de me restituer les notes que je leur avais confiées. . .) Le premier exposé (i.e. the introductory Exposé) consistait en un vaste "tour d’horizon" de ce qui avait été accompli dans le séminaire précédent SGA 4, en ce qui concerne le formalisme cohomologique étale et ses relations à divers autres contextes. Le deuxième exposé (this is the Exposé 0 which I referred to) développe en long et en large le formalisme "abstrait" des six variances. Il y a un formulaire essentiellement complet, mais sans effort encore pour cerner les compatibilités entre isomorphismes canoniques. (C’était là une tâche de nature plus technique, inutile à un moment où je tenais avant tout à "faire passer" ce yoga de dualité, dont je sentais bien tout la force.) Inutile de dire qu’il n’y a trace dans l’édition-Illusie ni de l’un ni de l’autre exposé.
I think Grothendieck wished to express the formalism in a general framework of derived and triangulated categories, just like he did with his Riemann-Roch theorem in SGA 6.
The theory of derived and triangulated categories was developed by Verdier in his thesis, but he decided against publishing it. Nevertheless, an embryonic version appeared as an État 0, from where (I think) they took the part about derived categories in SGA 4 1/2. The full work only saw the light of day in 1996 (posthumously), thanks to Illusie's efforts.
You can also find here a handwritten "cheatsheet" by Grothendieck, summarizing the "formalisme des six opérations".
