Let $X \in \mathbb{R}^{n \times n}$$X \in \mathbb{R}^{m \times n}$ be the matrix with columns $x_1$, $\ldots$, $x_n$. Then your matrix $A$ can be written as $$ A = X^TX(nI - J), $$ where $J$ is the $n \times n$ matrix with every entry equal to $1$. The matrices $X^TX$ and $nI-J$ are both positive semidefinite and thus have non-negative real eigenvalues. It follows that the same is true of the eigenvalues of their product $A$, so $I + A$ has all of its eigenvalues real and at least $1$, so it is invertible.
It's worth pointing out that this doesn't depend at all on the $x_i$'s being distinct: $I + A$ is invertible regardless.
As for an explicit formula for the inverse, I doubt that one exists, since too many matrices can be written in this form: every positive semidefinite matrix can be written in the form $X^TX$, and multiplying by $nI - J$ doesn't do much to eigenstuff other than ensure than one eigenvalue equals $0$. With so few restrictions, this is too close to just asking for a formula for the inverse of any matrix with positive eigenvalues.
