So maybe everything I'm about to say you already know, so apologies if I'm teaching my grandmother to suck eggs.
This is discussed a bit at the end of a paper of Fontaine "Representations $\ell$-adiques potentiellement semistables" (section 2.4) where he describes a notion of independence that (kind of) doesn't depend on base changing to $\mathbb{C}$. It's a bit hard to find, but it's in the "Periodes $p$-adiques" Asterisque volume (for some reason, I can't get onto MathSciNet today, so I can't give you better links).
UPDATE: The Imperial WiFi is now working properly, here's the MathSciNet link
From my understanding of it, he claims $\ell$-independence (in a reasonably strong form) for abelian varieties, curves (without conditions on the reduction), and anything with good reduction. As has been mentioned, the good reduction case is Deligne. (Also, as usual, the abelian variety case implies the case $i=1$ for any $X$, so you know $i=0,1,2d-1,2d$ when $X$ has dimension $d$.)
I'm not exactly sure how the proof goes for abelian varieties in general, but with semistable reduction this basically follows from the explicit description of the weight/monodromy filtration in SGA 7, Expose IX, together with the fact mentioned by Fontaine that 'compatibility' of Weil-Deligne reps follows from equality of characters on the graded pieces of monodromy.
I'm not sure about other cases you mention, like potentially good or potentially semistable reduction. If you knew the weight monodromy conjecture, then (I think) the weight spectral sequence would give you $\ell$-independence, again using the above check for 'compatibility' and the fact that you know $\ell$-independence for the $E_1$-page. But again, I'm not sure how to pass from semistable to potentially semistable.
Finally, when $X$ has dimension $2$, since you know $i=0,1,3,4$, if you know independence results for the whole cohomology $H^*_\mathrm{et}(X/\mathbb{Q}_\ell)$ then you know it for $i=2$. There are results to this effect here: