# Convex cones: strict separation

Consider two closed convex cones $A$ and $B$ in $\mathbb{R}^3$. Assume that they are convex even without zero vector, i.e. $A \setminus \{0\}$ and $B \setminus \{0\}$ are also convex (it helps to avoid weird cases like a plane being convex cone). Suppose that they do not have common directions, i.e. $A \cap B = \{0\}$. It seems that in this case there must exist a strict separating plane, i.e. there exists some unit vector $d$ such that $d \cdot a > 0$ for $a \in A \setminus \{0\}$ and $d \cdot b < 0$ for $b \in B \setminus \{0\}$.

I have tried to prove it myself, but to no avail. If we take a unit sphere $S$, then we may consider convex sets $\mathcal{H}(A \cap S)$ and $\mathcal{H}(B \cap S)$. They are compact and disjoint, so perpendicular bisector between the pair of closest points should separate these sets. If these two closest points are both on unit sphere $S$, then the bisector passes through origin and everything is OK. But in other cases the bisector is not helpful.

After some searching, I have found several topics which may be related to the question:

1. Hahn–Banach separation theorem gives separating plane for convex sets. However, the plane provided may touch both cones.
2. Farkas's lemma also provides a separating plane for cones. But in classical formulation one cone must be a ray, and the other one must be cone hull of finite number of vectors. Also, the separating plane constructed is nonstrict for one of the cones.
3. $A \cap S$ and $B \cap S$ should be geodesically convex on the unit sphere $S$. Since they do not intersect, there might be a strict separating big circle between them. I'm not even sure how to treat this sort of objects though.

I would be glad to see either a reference to literature or a simple proof.

I think you may be misreading the Hahn-Banach theorem in this case, as it should give you a strict separating plane here. Anyway, you can avoid this problem by enlarging the cones a bit before applying Hahn-Banach. This is no problem in the finite dimensional case.

Here's what I believe is a direct proof: Find a codimension-1 hyperplane $H$ such that the cross-section $A\cap H$ is compact and convex (such and $H$ should exist by your hypotheses). The intersection $B \cap H$ will be closed and convex, though perhaps not compact. Treat $H$ itself as a linear space and apply Hahn-Banach to strictly separate $A\cap H$ from $B\cap H$ by a codimension-1 (in $H$) hyperplane $G$. Complete $G$ to a codimension-1 (in the ambient space) hyperplane passing through the origin, which will separate the cones $A\setminus \{0\}$ and $B\setminus \{0\}$ as you wanted.

In the infinite dimensional case, it is of course more difficult and requires stronger hypotheses (one of the cones has to be closed and the other have a compact cross-section) and there is some literature on this. For example:

Separation Properties of Convex Cones, by V. L. Klee, Jr. PAMS 6 313-318 (1955)

• Your reference fits perfectly! Obviously, $A' = A \cap -A = \{0\}$ for any cone I have. Hence Theorem 2.7 proves exactly what I need, even in a more general case. – stgatilov Oct 9 '15 at 14:28
• I'm not sure your direct proof works. Even if you can cut properly the cone $A$ with a plane $H$, this cutting plane may not even intersect the $B$ cone, so $G$ would separate a convex set from nothing. Take two opposite $45^{\circ}$ circular cones as $A$ and $B$, for example. – stgatilov Oct 9 '15 at 14:32
• As for Hahn-Banach theorem, I cannot see how to apply it directly (at least with wikipedia formulations). I think it is not possible, because some additional knowledge about cones is necessary to prove strict separation: simply having closed convex cones is not enough (recall a plane being a cone), it is necessary to know that opposite vectors are not present in each of the cones, I guess. – stgatilov Oct 9 '15 at 14:37
• The idea with enlarging the cones is great, I really missed it! The cones can be enlarged by small angle. It is easy to show that enlarged cones are disjoint, also it is easy to find a separating plane. However, the main difficulty remains in proving that a cone can be enlarged without losing convexity. And this question is not so clear, I'm afraid. – stgatilov Oct 9 '15 at 14:41
• You need the existence of a well-behaved cross-section for the individual cones as a hypothesis, no way around this. Another way to phrase this is to ask the cone not to contain any affine subspace of positive dimension (a salient cone). Non-salient cones non-trivially intersect every codim-1 subspace, so cannot be separated from anything. – Igor Khavkine Oct 10 '15 at 14:17