(1) Is category theory the new language of mathematics, or recently the more preferred language?
Category theory has been proposed in 1940s and started taking over algebraic geometry and topology first in 1970s, and its application has only grown from there.
Whether it is the preferred language depends on which field of mathematics you are thinking about.
Generally, fields with an algebraic flavor prefer category theory. Examples include algebraic geometry, algebraic topology, category theory (duh), algebraic set theory, topological quantum field theory (new branch in physics), type theory.
Fields with an analysis/calculus flavor prefer set theory. Examples include basic calculus, differential equations, differential geometry, functional analysis, probability, set theory (duh).
Finally, fields with a geometric flavor seem to go via a third way, based on geometric intuition and "self-evident postulates". Examples include geometric topology (especially when low-dimensional), knot theory, general physics (I count it as the most important branch of mathematics), and some new fields that have "synthetic" in their names, like synthetic differential geometry.
(2) Recognizing that set theory can be articulated or founded through category theory (the text from Rosebrugh and Lawvere), is category theory now seen as the foundation of mathematics?
General mathematicians don't care much about the foundation. Currently there is a feeling that anything that has been worked on for a long time is sound. Mathematics is now seen as a massive graph of math-nuggets that connect to each other, floating in a vacuum with no special nuggets considered as "the true foundation".
However, this does not mean that all math-nuggets are equally significant, or foundation-worthy. Generally, significance is measured by these characteristics:
- How densely a nugget is connected to other nuggets;
- How "deep" the connections are;
- How fashionable it is (synthetic geometry is not fashionable now, but was extremely fashionable 2000 years ago);
- How close to physical reality it is (this makes basic calculus more significant than theory of prime numbers).
For a good discussion of what makes a nugget "significant", read Hardy's A Mathematician's Apology, starting at section 11.
As for foundation-worthiness, Maddy's What Do We Want a Foundation to Do? is a great place to learn about the details. I think that in short, some nuggets are better suited for foundation if they satisfy the following criteria:
- Can encode the things that mathematicians want to work with. This is analogous to a programming language being "Turing complete". A foundation must be "mathematician complete" or get close to it.
- Can encode elegantly. This is less objective but also very important. In the algebraic-flavored fields, category theory wins over set theory in this aspect.
- Can be checked mechanically, that is, it is good for formal verification. This is not yet a very important consideration, but is something univalent foundations and other type-theory foundations are explicitly trying to do well. For more, this Quanta essay is good for a start. For another, A computer-generated proof that nobody understands
(3) Is the choice between category theory language and set theory language maybe just depending on the field of mathematics, i.e. some fields tend to prefer set theory, others category theory?
What I said.
I have changed my mind about the foundation-worthiness of set theory. My new view is that set theory is inappropriate as a foundation of mathematics done by non-set-theorists.
Saunders Mac Lane (co-discoverer of category theory) was very keen on philosophical issues. His view is that set theory is not a good foundation to "real mathematics", since it does not encode the language of math as done by non-set-theorists. Category theory does the job much better. He thought that ZFC is particularly "inappropriate" because it is too strong, and just like how we should have "appropriate technology", mathematicians should use "appropriate foundations".
From To the Greater Health of Mathematics:
Set theory is not the only viable foundation for mathematics. It is striking that Smorynski, for all his extensive knowledge, appears to ignore the Lawvere idea of replacing set-membership by composition of functions as a primitive notion in foundations; this is an idea now explicitly available in the theory of an elementary topos. This approach to foundations has the advantage that it is closer to the actual practice of mathematics. It emphasizes the idea that the study of foundations is not restricted to consistency and proof-theoretic strength, but also covers analysis of the conceptual structure of mathematics, poorly reflected in the usual set theoretic translation.
There is a category-theoretic construction of ZFC set theory, called ETCS (Elementary Theory of the Category of Sets), Todd Trimble wrote about ETCS extensively in a series of essays, and observed that ZFC is ridiculously "strong" because of the extensionality axiom. This strength is inappropriate, as real mathematics does not require it, and it creates complications.
The deep meaning of the extensionality axiom is that a “set” $S$ is uniquely specified by the abstract structure of the tree of possible backward evolutions or behaviors starting from the “root set” $S$. This gives some intuitive but honest idea of the world of sets according to the ZFC picture: sets are tree-like constructions. The ZFC axioms are very rich, having to do with incredibly powerful operations on trees, and the combinatorial results are extremely complicated.
ZFC is an axiomatic theory (in the language of first-order logic with equality), with one basic type V and one basic predicate ∈ of binary type V×V, satisfying a number of axioms. The key philosophic point is that there is no typed distinction between “elements” and “sets”: both are of type V, and there is a consequent very complicated dynamical “mixing” which results just on the basis of a short list of axioms: enough in principle to found all of present-day mathematics!
My own reaction is that ZFC is perhaps way too powerful! For example, the fact that ∈ is an endo-relation makes possible the kind of feedback which can result in things like Russell’s paradox, if one is not careful. Even if one is free from the paradoxes, though, the point remains that ZFC pumps out not only all of mathematics, but all sorts of dross and weird by-products that are of no conceivable interest or relevance to mathematics.
Update: John Baez gave a good alternative word for "foundation" in Foundations of Mathematics:
Personally I don’t think the metaphor of “foundations” is even appropriate for this approach. I prefer a word like “entrance”. A building has one foundation, which holds up everything else. But mathematics doesn’t need anything to hold it up: there is no “gravity” that pulls mathematics down and makes it collapse. What mathematics needs is “entrances”: ways to get in. And it would be very inconvenient to have just one entrance.