[EDIT: I wrote my original answer when I was in a bit of a hurry. I have now expanded my answer.]
The short answer is no. The simplest way to see that $C^A$ cannot possibly depend only on $C$ and $A$ as sets of strings is the following spurious argument that has confused generations of students. Assume that $P=NP$. Then for all oracles $A$, $P^A = NP^A$. But by Baker–Gill–Solovay, we know that there exists an oracle $A$ such that $P^A \ne NP^A$. This is a contradiction. Hence $P\ne NP$. Q.E.D., and I await my $1 million check.
The standard notation $C^A$ is an abuse of notation. A class per se cannot be relativized; one must specify a model of computation and provide that model of computation with access to an oracle. Choosing different models of computation, or different oracle access mechanisms, may give you different relativizations. For example, space-bounded classes are notoriously tricky to relativize, as explained in this paper by Hartmanis et al. The trouble is that it's not obvious what the "right" oracle access mechanism is in certain cases.
Having said that, I think it is a fascinating open-ended problem to try to "formalize" the concept of relativization in a way that would allow one to rigorously prove statements of the form, "such-and-such a type of argument relativizes and therefore cannot (for example) separate $P$ from $NP$." Though this is a very tempting idea, there is still no fully satisfactory theory along these lines. Probably the best attempt in this direction so far is the unpublished manuscript Relativizing versus nonrelativizing techniques: the role of local checkability, by Arora, Impagliazzo, and Vazirani, but this is really just a first attempt. For example, in his paper The role of relativization in complexity theory, Fortnow argues that there are certain important limitations to the Arora–Impagliazzo–Vazirani model. Fortnow's paper is recommended reading if you want to work on this open problem yourself.