We have the unit sphere $S^2$ in $\mathbb{R}^3$ and two points, $X$ and $Y$ on the surface of the sphere. Then, a function is defined for any point $P$ inside of the unit ball as:

$$f(P) = R\,d(P, XY)$$

where $d(P,XY)$ is the Euclidean distance from the line containing $XY$, and $R$ is the radius of the small circle of the sphere in the plane containing $XYP$. One can notice that for a point $P'$ on the surface of the sphere, the function corresponds to $f(P') = \frac{1}{2} |P'X| |P'Y|$.

I can obtain values of the function numerically by calculating $R$ as the distance of $XYP$ from the origin, but it does not seem to simplify to any useful closed-form expression. Suppressing any one of the dimensions, a plot of the function (for a choice of $X,Y$) looks like this:

where the minimum is reached for the line $XY$, the function is constant on lines parallel to $XY$, and it is visibly convex.

How to show this convexity? The distance is obviously a convex function, but $R$ is not convex.