Yes _if_ all matrices have the same rank. Your $x^*(\delta)$ is $A(\delta)^+b$ where $A^+$ is the _Moore-Penrose pseudoinverse_ of $A$, which depends continuously on $A$ if (and only if) restricted to matrices of the same rank.

For the more general case, I don't know.