This is not an answer, but it was getting too long to be a comment. Also, I'm not familiar with such problems, so hopefully someone will be able to give you a complete and more satisfying answer and some references. Here are some general remarks. It's a bit messy and mostly guesswork :S

If $|A^{-1}b|<1$ there might be several solutions, i.e. several $x$ minimizing $\varphi(x):=\frac{1}{2}\langle x|Ax \rangle+\langle b|x \rangle$ (I added $\frac{1}{2}$ for convenience only), precisely, if you call $\lambda_1>\dots>\lambda_p>0$ the distinct eigenvalues of $A$ and you have $e_1,\dots, e_p$ an associated orthonormal frame and $b$ such that $$|A^{-1}b|<1~\mathrm{and}~b\in\mathrm{Span}(e_1,\dots,e_r)\setminus\mathrm{Span}(e_1,\dots,e_{r-1})$$

I would expect the set of minimizing $x$ to be a sphere of dimension $p-r-1$ and a point when $r=p$

If $|A^{-1}b|>1$, a picture convinces me there is only one solution to your problem (and the "projection onto a convex closed set"-theorem in inner product spaces proves it). If $x_0$ is a solution, then we have to have that the differential of $\varphi$ vanishes at $x_0$ which translates into $\langle Ax_0|- \rangle+\langle b|- \rangle$ vanishing on the orthogonal of $x_0$ i.e. there exists some $\mu\in\mathbb{R}$ such that $Ax_0+b=\mu x_0$, and again, a picture suggests that we have to look for solutions with $\mu < 0 $. Then there is indeed only one solution, because if we look for $x_0\in\mathbb{S}^{n-1}$ satisying above equation, we can write down $x_0=-(A-\mu)^{-1}b$ because the matrix is invertible. The norm of above vector is stricly increasing with $\mu$ and equals $|A^{-1}b|>1$ at $\mu=0$ and tends to $0$ as $\mu$ tends to $-\infty$. So there'll be only one $\mu<0$ such that $-(A-\mu)^{-1}b$ has unit length. But finding that given $\mu$ requires solving the equation $\frac{b_1^2}{(\lambda_1-\mu)^2}+\dots+\frac{b_p^2}{(\lambda_p-\mu)^2}=1$

So if $|A^{-1}b|>1$, you can find the solution as long as you can find the negative $\mu$ that satisfies the above equation...

This is not an answer, but it was getting too long to be a comment. Also, I'm not familiar with such problems, so hopefully someone will be able to give you a complete and more satisfying answer and some references. Here are some general remarks. It's a bit messy and mostly guesswork :S

If $|A^{-1}b|<1$ there might be several solutions, i.e. several $x$ minimizing $\varphi(x):=\frac{1}{2}\langle x|Ax \rangle+\langle b|x \rangle$ (I added $\frac{1}{2}$ for convenience only), precisely, if you call $\lambda_1>\dots>\lambda_p>0$ the distinct eigenvalues of $A$ and you have $e_1,\dots, e_p$ an associated orthonormal frame and $b$ such that $$|A^{-1}b|<1~\mathrm{and}~b\in\mathrm{Span}(e_1,\dots,e_r)\setminus\mathrm{Span}(e_1,\dots,e_{r-1})$$

I would expect the set of minimizing $x$ to be a sphere of dimension $p-r-1$ and a point when $r=p$

If $|A^{-1}b|>1$, a picture convinces me there is only one solution to your problem (and the "projection onto a convex closed set"-theorem in inner product spaces proves it). If $x_0$ is a solution, then we have to have that the differential of $\varphi$ vanishes at $x_0$ which translates into $\langle Ax_0|- \rangle+\langle b|- \rangle$ vanishing on the orthogonal of $x_0$ i.e. there exists some $\mu\in\mathbb{R}$ such that $Ax_0+b=\mu x_0$, and again, a picture suggests there are preciseyl two solutions and that we have to look for the solutions with $\mu < 0 $. Then there is indeed only one solution, because if we look for $x_0\in\mathbb{S}^{n-1}$ satisying above equation, we can write down $x_0=-(A-\mu)^{-1}b$ because the matrix is invertible. The norm of above vector is stricly increasing with $\mu$ and equals $|A^{-1}b|>1$ at $\mu=0$ and tends to $0$ as $\mu$ tends to $-\infty$. So there'll be only one $\mu<0$ such that $-(A-\mu)^{-1}b$ has unit length. But finding that given $\mu$ requires solving the equation $\frac{b_1^2}{(\lambda_1-\mu)^2}+\dots+\frac{b_p^2}{(\lambda_p-\mu)^2}=1$

So if $|A^{-1}b|>1$, you can find the solution as long as you can find the negative $\mu$ that satisfies the above equation...

This is not an answer, but it was getting too long to be a comment. Also, I'm not familiar with such problems, so hopefully someone will be able to give you a complete and more satisfying answer and some references. Here are some general remarks. It's a bit messy and mostly guesswork :S

If $|A^{-1}b|<1$ there might be several solutions, i.e. several $x$ minimizing $\varphi(x):=\frac{1}{2}\langle x|Ax \rangle+\langle b|x \rangle$ (I added $\frac{1}{2}$ for convenience only), precisely, if you call $\lambda_1>\dots>\lambda_p$ the distinct eigenvalues of $A$ and you have $e_1,\dots, e_n$ an associated orthonormal frame and $b$ such that $$|A^{-1}b|<1~\mathrm{and}~b\in\mathrm{Span}(e_1,\dots,e_r)\setminus\mathrm{Span}(e_1,\dots,e_{r-1})$$

I would expect the set of minimizing $x$ to be a sphere of dimension $p-r-1$ and a point when $r=p$

If $|A^{-1}b|>1$, a picture convinces me there is only one solution to your problem. If $x_0$ is a solution, then we have to have that the differential of $\varphi$ vanishes at $x_0$ which translates into $\langle Ax_0|- \rangle+\langle b|- \rangle$ vanishing on the orthogonal of $x_0$ i.e. there exists some $\mu\in\mathbb{R}$ such that $Ax_0+b=\mu x_0$, and again, a picture suggests that we have to look for solutions with $\mu < 0 $. Then there is indeed only one solution, because if we look for $x_0\in\mathbb{S}^{n-1}$ satisying above equation, we can write down $x_0=-(A-\mu)^{-1}b$ because the matrix is invertible. The norm of above vector is stricly increasing with $\mu$ and equals $|A^{-1}b|>1$ at $\mu=0$ and tends to $0$ as $\mu$ tends to $-\infty$. So there'll be only one $\mu<0$ such that $-(A-\mu)^{-1}b$ has unit length. But finding that given $\mu$ requires solving the equation $\frac{b_1^2}{(\lambda_1-\mu)^2}+\dots+\frac{b_p^2}{(\lambda_p-\mu)^2}=1$

So if $|A^{-1}b|>1$, you can find the solution as long as you can find the negative $\mu$ that satisfies the above equation...

This is not an answer, but it was getting too long to be a comment. Also, I'm not familiar with such problems, so hopefully someone will be able to give you a complete and more satisfying answer and some references. Here are some general remarks. It's a bit messy and mostly guesswork :S

If $|A^{-1}b|<1$ there might be several solutions, i.e. several $x$ minimizing $\varphi(x):=\frac{1}{2}\langle x|Ax \rangle+\langle b|x \rangle$ (I added $\frac{1}{2}$ for convenience only), precisely, if you call $\lambda_1>\dots>\lambda_p>0$ the distinct eigenvalues of $A$ and you have $e_1,\dots, e_p$ an associated orthonormal frame and $b$ such that $$|A^{-1}b|<1~\mathrm{and}~b\in\mathrm{Span}(e_1,\dots,e_r)\setminus\mathrm{Span}(e_1,\dots,e_{r-1})$$

I would expect the set of minimizing $x$ to be a sphere of dimension $p-r-1$ and a point when $r=p$

If $|A^{-1}b|>1$, a picture convinces me there is only one solution to your problem (and the "projection onto a convex closed set"-theorem in inner product spaces proves it). If $x_0$ is a solution, then we have to have that the differential of $\varphi$ vanishes at $x_0$ which translates into $\langle Ax_0|- \rangle+\langle b|- \rangle$ vanishing on the orthogonal of $x_0$ i.e. there exists some $\mu\in\mathbb{R}$ such that $Ax_0+b=\mu x_0$, and again, a picture suggests that we have to look for solutions with $\mu < 0 $. Then there is indeed only one solution, because if we look for $x_0\in\mathbb{S}^{n-1}$ satisying above equation, we can write down $x_0=-(A-\mu)^{-1}b$ because the matrix is invertible. The norm of above vector is stricly increasing with $\mu$ and equals $|A^{-1}b|>1$ at $\mu=0$ and tends to $0$ as $\mu$ tends to $-\infty$. So there'll be only one $\mu<0$ such that $-(A-\mu)^{-1}b$ has unit length. But finding that given $\mu$ requires solving the equation $\frac{b_1^2}{(\lambda_1-\mu)^2}+\dots+\frac{b_p^2}{(\lambda_p-\mu)^2}=1$

So if $|A^{-1}b|>1$, you can find the solution as long as you can find the negative $\mu$ that satisfies the above equation...