The actual category $A$ does not matter. The only thing that matters is the "shape" of $A$. We define the limit to be the hom-set $Hom_{Set^A}(*,F)$ from the terminal functor into $F$. It does depend on the set-theoretic universe, and I've glossed over those details, but you can be reasonably certain that it really doesn't matter once we pick $Set$ to be large enough.
If you mean by "set-theoretic universe" that we pick different axioms for our sets, then it obviously depends a lot on this. $A$ is defined in terms of the category of sets.
There is no problem with category theory. The problem is that you don't understand what kinds of questions we ask and answer using it. You are asking questions akin to "Let X be a set. What is the cardinality of X?" We can define the cardinality just fine. We cannot determine it without more information. In fact, since we know that $X$ is a set, we know that its cardinality is a cardinal denoted by $|X|$, but pending more information, we cannot say anything more.