# Notion of smoothness for set-valued functions

Is there a way of talking about continuity and smoothness for set valued functions? More precisely, consider $M$ and $N$ topological/smooth manifolds, and let $f$ a function that associates to each point $p\in M$ a subset $f(p) \subset N$ (I haven't made any assumptions on what target sets are allowed, but feel free to discuss cases where some restrictions are required). Is there a meaningful/canonical way of saying that $f$ is continuous or smooth?

For my particular application in mind, $M$ is a smooth manifold, and $f$ associates to each $p\in M$ an open, convex cone inside $T_pM$.

Edit: I should clarify that a convex cone $K$ in some real vector space $V$ is a subset such that

(a) conic: for any $v\in K$ and $r\in \mathbb{R}_+$ $\implies rv \in K$.

(b) convex: for any $v,w\in K$ and $a,b \in \mathbb{R}_+$ $\implies av + bw \in K$

It is open if $K$ is an open subset of $V$, so in particular open cones do not contain the origin.

-
In your particular example it seems that the usual condition of local triviality (like in smooth sub-bundles) might be an option. – Michael Bächtold Sep 8 '10 at 14:29
I thought about that, but ran into the following problem: for a smooth vector subbundle $D\subset F$, local triviality can be implemented by $D$ being pointwise a vector subspace, which is compatible with the linear structure on $F$. In my case, ideally if I use a condition like that it should be compatible with the linear structure in some way, but one cannot always map an open convex cone to another using a linear map (think one of them being the open half space). Which is why I am trying to get at something using less structure. – Willie Wong Sep 8 '10 at 14:55
but you can forget about the linear structure and just treat the tangent bundle as a fiber bundle. – Michael Bächtold Sep 8 '10 at 15:12
Willie, it seems to me that the "right" definition depends a lot on what you need this for. – Deane Yang Sep 8 '10 at 17:03
In your application, are your cones' boundaries are smooth? If so, you can say that the map is smooth if the union of these boundaries is a smooth submanifold of $TM$. This is consistent with smoothness in Finsler geometry (where one can say that Finsler metric = family of norms = family of convex bodies in the fibers of the tangent bundle). – Sergei Ivanov Sep 8 '10 at 18:10

My idea is that if we want to compare $f(p)$ and $f(q)$ for nearby points $p$ and $q$, then we need to be able to put $f(p)$ and $f(q)$ into the same space. To do this, I'm going to assume that $M$ is a finite-dimensional Riemannian manifold, so that we can make use of a connection $\nabla$ on $M$.

For all $p$, let $f(p)$ be an open cone of $T_p M$. Let $U_p M$ denote the unit sphere in $T_p M$, and define $$g(p) = f(p) \cap U_p M.$$ Since $f(p)$ is a cone, it is the linear span of $g(p)$. Thus, any smoothness on $g$ will apply to $f$ as well.

Let $w \in T_p M$, and define the covariant derivative of $g(p)$ in the direction $w$ by $$\nabla_w g(p) := \{ \nabla_w v \}_{v \in g(p)}.$$

Let $\gamma$ be a smooth curve on $M$ with $\gamma(0) = p$ and $\dot \gamma(0) = w$. Define the parallel transport of $g(p)$ along $\gamma(t)$ by $$\nabla_{\dot\gamma(t)} g(p) := \{ \nabla_{\dot\gamma(t)} v \}_{v \in g(p)}.$$ That is, the parallel transport of the set $g(p)$ is given by transporting each vector in $g(p)$ along the curve $\gamma(t)$.

Now, both $\nabla_{\dot\gamma(t)} g(p)$ and $g(\gamma(t))$ are open subsets of the unit tangent space $U_{\gamma(t)} M$ at the point $\gamma(t)$. If the set-valued function $g$ is to be smooth, then these two sets should be comparable.

Let $\operatorname{Vol}_q$ denote the (finite) volume measure on the unit sphere $U_q M$ at the point $q \in M$, and let $\Delta$ denote the symmetric difference of two sets. Let us say that $g$ is smooth at $p$ in the direction $w$ if $$\operatorname{Vol}_{\gamma(t)} \left( \nabla_{\dot\gamma(t)} g(p) ~\Delta~ g(\gamma(t)) \right) = O(t)$$ for all smooth functions $\gamma$ with $\gamma(0) = p$ and $\dot \gamma(0) = w$. If $g$ is smooth at all points in all directions, then we shall say it is smooth on $M$. Consequently, we shall say that $f$ is smooth if $g$ is smooth.

-
I rather like this idea. I am wondering if perhaps the use of the Levi-Civita connection is unnecessary. It seems to me that a change to a different torsion-free connection should only introduce an O(t) error term. Ideally the notion of smoothness should be independent of the actual choice of connection. – Willie Wong Sep 8 '10 at 18:25
If I pare this down a bit, it seems to be equivalent to what Sergei Ivanov proposed in the comments above. – Willie Wong Sep 8 '10 at 18:33
Willie, I'm certain that the full Levi-Civita connection is unnecessary-- it's just the one that most readily came to mind. I would not at all be surprised if this is equivalent to a Finsler-geometry notion of smoothness. – Tom LaGatta Sep 8 '10 at 21:49
I'm accepting this one since it is probably the easiest to implement in the case I am considering; also because Deane and Sergei both gave essentially the same answer in their comments. Thanks to everyone who helped out. Cheers. – Willie Wong Sep 9 '10 at 13:09

Three outcomes of a short brainstorming (all inspired by Algebraic Geometry):

1. I second Todd Trimble's comment to your question (it deserves to appear in an answer, so I repeat it here): Synthetic differential geometry gives you a way to talk about "manifolds of subsets", and you have sort of an automatic smoothness built in. But getting into this probably takes you on a long detour...

2. The Algebraic Geometer's way of treating many-valued functions is, very sloppily: Identify functions $X \rightarrow Y$ with their graphs, i.e. subvarieties $\Gamma \subseteq X \times Y$ with $pr_X(\Gamma)=X$ (defined on all of X) and $|pr_X^{-1}(x) \cap \Gamma|\leq 1 \ \forall x \in X$ (single-valued), then drop the second requirement. So maybe you can look at smooth submanifolds of $M \times TM$ subject to the conditions you want?

3. The most reasonable, I would say, and close to the previous: Go to differentiable stacks. For your particular situation you could look at the classifying stack of cones inside tangent bundles: This would be the category fibred in groupoids over the site of differentiable manifolds which has objects $(M,C)$ with $M$ a smooth manifold, $C \subseteq TM$ a submanifold (smooth, except for the tip of the cone) of the total space of the tangent bundle such that each fiber is a cone in $T_x(M)$. Morphisms $(M,C) \rightarrow (M',C')$ should probably be smooth morphisms $f:M \rightarrow M'$ such that $C$ is the pullpack of $C'$ along $Tf$, the differential of $f$. Now a map of differentiable stacks from a manifold $N$ into this stack is the same as a smoothly varying choice of cones in the fibers of the tangent bundle $TN$. The same technique should work with other set-valued maps and is very flexible if you want to modify conditions.

-
Some very interesting ideas. I will investigate along these lines a bit in the future. Thanks for the suggestions. – Willie Wong Sep 9 '10 at 13:08

One possible way is to demand that the target set $f(p)$ is a compact set and use Hausdorff distance on the set of compact subsets of $N$ (this aplies to your application by considering the proyective space of $T_p M$).

In the particular aplication, If the cone of $T_p M$ is defined by a subspace of $T_p M$ together with an angle and we assume that continuity implies that the dimension of the cones is constant (this would be given using the Hausdorff distance in the proyective space as above), you can test "smoothness" by considering the map from $M$ to the bundle of grasmannians times angle which is a diferentiable manifold and gives a meaningful way of saying that $f$ is smooth.

However, I believe it should be accepted that the cones "colapse" and decrease its dimension (even if they are never trivial, they can "change their dimension"). For this, the only way I imagine is to consider subsets $(T_pM)^n \times \mathbb{R}$ and consider the cone as the one generated by the $n$ vectors and with angle the value in $\mathbb{R}$ (clearly, there is no canonical way of considering this function, one should say that the map is smooth if there exists a function $g$ to $(T_pM)^n \times \mathbb{R}$ which defines $f$).

-
I don't understand the claim that "open convex cone of $T_pM$ is defined by a subspace together with an angle". I do not require my cone to be a quadratic cone (in which case your argument is very nice). I want cone in the sense that if $v$ is in the set, then $rv$ is also in the set for any positive real $r$. In particular, associated to any open, convex shape in $\mathbb{S}^{d-1}$ is an open convex cone in $\mathbb{R}^d$. The open condition does, however, ensures that the cones cannot collapse. – Willie Wong Sep 8 '10 at 15:01
I will edit that. I misunderstood the definition. – rpotrie Sep 8 '10 at 15:16

Here are some random thoughts. If your cones are polyhedral cones, then maybe you can do something like the following:

Suppose $M$ is $n$ dimensional. Let $Gr_{n-1}(TM)$ be the Grassmannian bundle over $M$ such that the fiber over $p$ is the Grassmannian of $(n-1)$-planes in $T_p M$. Let $Gr_i$ be the product bundle $Gr_{n-1}(TM)^{i}$. Let $Gr$ be some appropriate colimit of the $Gr_i$'s. For example, you could take the colimit of the maps $Gr_i \to Gr_{i+1}$ given by $(P_1,\dots,P_i) \mapsto (P_1,\dots,P_i,P_i)$.

Then define a smooth/continuous/whatever polyhedral-cone-valued-function $f$ to be a smooth/continuous/whatever section $s_f$ of the bundle $Gr$ over $M$. The section $s_f$ assigns to the point $p$ the hyperplanes which form the faces of the polyhedral cone $f(p)$.

I guess this works if you have orderings on the faces of the polyhedral cones. If you don't have orderings, you could take symmetric products of the bundles instead of products.

-
Nice idea. Won't be particularly useful for my exact problem (sorry!) since my cones will not be polyhedral in general, but a nice idea still. – Willie Wong Sep 8 '10 at 18:28
If your cones are bounded by algebraic hypersurfaces, perhaps you could still do something similar... – Kevin H. Lin Sep 8 '10 at 19:24