These are good answers and I have nothing to add on the particular reasons. To sum it up I would say it was impossible to write a comprehensive Elements of Mathematics based on category theory, because it is impossible to write a comprehensive Elements of Mathematics at all. The brilliant attempt was very good for mathematics, I will maintain, but it could not really be done.
Categories and functors in the 1950s were developed only for immediate applications, and not as a general theory of structure. It was very much easier for Bourbaki to develop a general theory that did not work, than to unify a sprawling mass of working methods into one theory. And nothing was really going to work for their vast project anyway.
So far as I know no one talked about a general theory called "category theory" before biologist Robert Rosen in works like "A relational theory of the structural changes induced in biological systems by alterations in environment" Bull. Math. Biophys. 23 1961 165–171.