I can understand why $P^A = NP^A$ does not imply $P=NP$, $A$ can "contain" the powers of NP.
However, why does $P^B \neq NP^B$ not imply $P \neq NP$? It seems like if $P$ and $NP$ denote the same classes; then we should be able to arbitrarily substitute one for the other (as long as the only thing of interest is the computational model), and everything should stay the same.
More generally, why is $A^X \neq B^X$ not a poof for $A \neq B$ ?
I feel like there's a very fundamental piece of logic / reasoning I'm missing here.
EDIT: I understand the construction of the oracles A & B. However, I still don't understand why the existence of $B$ not prove that $P \neq NP$.