Well, you're almost right... Such a determinant is sometimes known as "lacunary vandermonde" determinant, although my internet searches have not given much. Lemma 4 in this paper states that the lacunary vandermonde determinant of variables $(x_1, \dots, x_n)$ missing the terms in $x_i^k$ is obtained by a product of the regular vandermonde of $(x_1, \dots, x_n)$ times a Viete sum.
You can work out for yourself how to prove this: it's not very difficult, but it's a good exercise. The point though is that, since you assumed your vandermonde was invertible, your lacunary determinant vanishes iff the Viete sum is zero. So there is a hypersurface of possible values for $(x_1, \dots, x_n)$ that cancel your lacunary determinant.