My understanding is if you have a homogeneous space $X = G/H$ for a reductive group $G$ and $X$ is normal and has an open $B$-orbit, for a Borel subgroup $B$ then you call $X$ spherical.
Someone asked me where 'spherical' came from and I had no idea. I asked a few more knowledgeable people and they also didn't know. So now I ask the same question here.
Marty is definitely correct about the origin of the terminology in the study of homogeneous spaces $G/H$ of reductive Lie groups. Due to the special example involving a quotient of special orthogonal groups the handy term *spherical" got attached to the subgroup $H$ or the homogeneous space. In turn, this was carried over to the situation of algebraic groups and similar quotients in the algebraically closed case, where spherical varieties could be characterized as those homogeneous spaces admitting a dense orbit under a Borel subgroup. For instance, look at the introduction of an influential paper by Brion-Luna-Vust which appeared in Inventiones 84 (1986) with the title Espaces homogènes sphériques. While the label spherical variety is convenient, it loses its literal sense in this generality.
Since I am being asked the same question repeatedly and since the given answers are not quite correct, I post another answer despite the thread being so old.
According to a talk by Domingo Luna around 1985, the term spherical variety is not derived from spheres, at least not directly. Firstly, spheres are way too atypical, e.g., their compactification theory is pretty pointless. Secondly, in invariant theory circles spheres are called quadrics anyway.
The true origin is a paper by Manfred Krämer from 1979 called "Sphärische Untergruppen in kompakten zusammenhängenden Liegruppen". Krämer had observed that one of the standard constructions for spherical functions generalizes from symmetric spaces to arbitrary homogeneous spaces $G/H$ with $G$ a compact Lie group. If $G/H$ contains not too many spherical functions (i.e., they commute under convolution) then he called $H$ a spherical subgroup and proceeded to classify them for simple $G$ in the paper mentioned above.
Around the same time it was realized (Vinberg-Kimelfeld) that Krämer's condition is equivalent to a Borel subgroup having an open orbit on the complexification of $G/H$, i.e., $G_{\mathbb C}/H_{\mathbb C}$ being spherical. Since Krämer's list does provide lots of non-trivial examples, Brion-Luna-Vust came up with their term.
So the implications are $$ \text{sphere}\Longrightarrow\text{spherical function}\Longrightarrow\text{spherical variety} $$ The first arrow this time makes sense since spherical functions were first seriously considered on spheres (Legendre and Gegenbauer polynomials).
Confirming Jim Humphreys' and Marty's answers, page 17 of (the English translation of) [Vinberg, È. B., Commutative homogeneous spaces and co-isotropic symplectic actions, MR1845642] contains:
In the simplest case of the two-dimensional sphere $S^2= SO(3)/SO(2)$ this result [the fact that the $SO(3)$-module $\mathbb{C}[S^2]$ is multiplicity-free] was known much earlier thanks to the Laplace spherical functions used in mathematical physics. This is the origin of the term “spherical homogeneous spaces”.
Note that if $X$ is an affine $G$-variety (with $G$ a complex reductive group), then $X$ has a dense orbit for a Borel subgroup $B \subset G$ if and only if $\mathbb{C}[X]$ is a multiplicity-free $G$-module.
