I'm going to assume that $M$ is a finite-dimensional Riemannian manifold, so that we can make use of a connection $\nabla$ on $M$. For all $p$, let $f(p)$ be an open, convex subset of $T_p M$. Let $U_p M$ denote the unit sphere in $T_p M$, and define $$g(p) = f(p) \cap U_p M.$$ Since $f(p)$ is a cone, it is the linear span of $g(p)$. Thus, any smoothness on $g$ will apply to $f$ as well.

Let $w \in T_p M$, and define the covariant derivative of $g(p)$ in the direction $w$ by $$\nabla_w g(p) := \{ \nabla_w v \}_{v \in g(p)}.$$

Let $\gamma$ be a smooth curve on $M$ with $\gamma(0) = p$ and $\dot \gamma(0) = w$. Define the parallel transport of $g(p)$ along $\gamma(t)$ by $$\nabla_{\dot\gamma(t)} g(p) := \{ \nabla_{\dot\gamma(t)} v \}_{v \in g(p)}.$$ That is, the parallel transport of the set $g(p)$ is given by transporting each vector in $g(p)$ along the curve $\gamma(t)$.

Now, both $\nabla_{\dot\gamma(t)} g(p)$ and $g(\gamma(t))$ are open subsets of the unit tangent space $U_{\gamma(t)} M$ at the point $\gamma(t)$. If the set-valued function $g$ is to be smooth, then these two sets should be comparable.

Let $\operatorname{Vol}_q$ denote the (finite) volume measure on the unit sphere $U_q M$ at the point $q \in M$, and let $\Delta$ denote the symmetric difference of two sets. Let us say that $g$ is smooth at $p$ in the direction $w$ if $$\operatorname{Vol}_{\gamma(t)} \left( \nabla_{\dot\gamma(t)} g(p) ~\Delta~ g(\gamma(t)) \right) = O(t)$$ for all smooth functions $\gamma$ with $\gamma(0) = p$ and $\dot \gamma(0) = w$. If $g$ is smooth at all points in all directions, then we shall say it is smooth on $M$. Consequently, we shall say that $f$ is smooth if $g$ is smooth.

My idea is that if we want to compare $f(p)$ and $f(q)$ for nearby points $p$ and $q$, then we need to be able to put $f(p)$ and $f(q)$ into the same space. To do this, I'm going to assume that $M$ is a finite-dimensional Riemannian manifold, so that we can make use of a connection $\nabla$ on $M$.

For all $p$, let $f(p)$ be an open cone of $T_p M$. Let $U_p M$ denote the unit sphere in $T_p M$, and define $$g(p) = f(p) \cap U_p M.$$ Since $f(p)$ is a cone, it is the linear span of $g(p)$. Thus, any smoothness on $g$ will apply to $f$ as well.

Let $w \in T_p M$, and define the covariant derivative of $g(p)$ in the direction $w$ by $$\nabla_w g(p) := \{ \nabla_w v \}_{v \in g(p)}.$$

Let $\gamma$ be a smooth curve on $M$ with $\gamma(0) = p$ and $\dot \gamma(0) = w$. Define the parallel transport of $g(p)$ along $\gamma(t)$ by $$\nabla_{\dot\gamma(t)} g(p) := \{ \nabla_{\dot\gamma(t)} v \}_{v \in g(p)}.$$ That is, the parallel transport of the set $g(p)$ is given by transporting each vector in $g(p)$ along the curve $\gamma(t)$.

Now, both $\nabla_{\dot\gamma(t)} g(p)$ and $g(\gamma(t))$ are open subsets of the unit tangent space $U_{\gamma(t)} M$ at the point $\gamma(t)$. If the set-valued function $g$ is to be smooth, then these two sets should be comparable.

Let $\operatorname{Vol}_q$ denote the (finite) volume measure on the unit sphere $U_q M$ at the point $q \in M$, and let $\Delta$ denote the symmetric difference of two sets. Let us say that $g$ is smooth at $p$ in the direction $w$ if $$\operatorname{Vol}_{\gamma(t)} \left( \nabla_{\dot\gamma(t)} g(p) ~\Delta~ g(\gamma(t)) \right) = O(t)$$ for all smooth functions $\gamma$ with $\gamma(0) = p$ and $\dot \gamma(0) = w$. If $g$ is smooth at all points in all directions, then we shall say it is smooth on $M$. Consequently, we shall say that $f$ is smooth if $g$ is smooth.