To turn a class into a category, you need a notion of morphisms between objects in the class. That's the long and short of it.
Consider for instance the class of infinite real series, say viewed as the set $S = \mathbb{R}^{\aleph_0}$ of sequences of real numbers. (There is often some notational and "ontological" confusion between the terms of an infinite series, its associated sequence of partial sums, and its sum, if it has one. Which one of these "is" the series? But such considerations are not relevant here and indeed are usually viewed as antithetical to the categorical point of view.) To get a category, you need to identify a set of morphisms between any two elements of this set. This can certainly be done in any number of ways -- for instance one could use the ordering induced from the standard ordering on $\mathbb{R}$ and the lexicographic ordering of the sequence, and then $S$ is a totally ordered set. We could then define a category by having $\operatorname{Hom}(s,t)$ to be a one point set if $s \leq t$ and the empty set otherwise (and then take the unique composition law of morphisms, defined when $s \leq t \leq u$).
But the question is: what does this category have to do with any aspect of the theory of
infinite series? Apparently nothing. You could create any number of other categories with underlying set $S$ but you run into the same problem: the very old and extremely well-developed area of mathematics which studies the convergence and divergence of real infinite series simply does not have anything evident to do with any notion of "morphisms" between infinite series.
Similarly for the other examples you mention. Categorical structure is a very fundamental kind of mathematical structure; it's a great way of thinking and unifies and conceptualizes the study of many kinds of mathematical objects in highly disparate fields. But it doesn't explain everything, and it is frankly a bit weird to think it should.