Indeed, as you suspect, the answer is no. For instance, take the link $L$ obtained from the Hopf link by doing a $(2,1)$-cable of one component and a Whitehead double of the other.
A way of telling them apart is to look at the JSJ decomposition of the complement: each piece contains one component, and one component is Seifert fibred, while the other is hyperbolic.
A reflection of this fact is that if you twist each of the components with respect to the other, you obtain different knots. (By "twist" here I essentially mean "do $\pm1$-surgery along one component.)
I also think that the 2-variable Alexander polynomial (or branched covers, coloured signatures...) should do the trick.