Yes if all matrices have the same rank. Your $x^*(\delta)$ is $A(\delta)^+b$ where $A^+$ is the Moore-Penrose pseudoinverse of $A$, which depends continuously on $A$ if (and only if) restricted to matrices of the same rank.

For the more general case, I don't know.

Yes if all matrices have the same rank. Your $x_*(\delta)$ is $A(\delta)^+b$ where $A^+$ is the Moore-Penrose pseudoinverse of $A$, which depends continuously on $A$ if (and only if) restricted to matrices of the same rank.

For the more general case, I don't know.