7
$\begingroup$

Fix $n,k\in \mathbb{N}_+$. Let $\mathcal{H}$ be a set of functions from $\mathbb{R}^n$ to $\mathbb{R}$ with finite VC-dimension $d\in \mathbb{N}$. Let $\mathcal{H}_k$ denote the set of maps of the form $$ f = \sum_{n=1}^{k_0}\, f_n\, I_{C_n} $$ where $k_0\in \mathbb{N}$ with $k_0\le k$, $f_1,\dots,f_{k_0}\in \mathcal{H}$, $I_A$ is the indicator function of a measurable set $A\subseteq \mathbb{R}^n$, and $C_1,\dots,C_{k_0}$ be the Voronoi cells corresponding to some distinct $p_1,\dots,p_{k_0}\in \mathbb{R}^n$; i.e. for $i=1,\dots,k_0$ we recursively define $$ C_i = \hat{C}_i\setminus \bigcup_{j<i} C_j $$ where for $i=1,\dots,k_0$ $$ \hat{C}_i = \{x\in \mathbb{R}^n:\, \|x-p_i\|= \min_{j=1,\dots,k_0}\,\|x-p_j\|\}. $$ Note: One may reorder the sets WLOG (breaking ties in a different manner when assigning equidistance points to one of the Voronoi cells).

Update: As pointed out by Gabe Conant, if the $C_i$ are not bonafide partitions, then the VC dimension of this class may degenerate.

Is it true that the VC-dimension of $\mathcal{H}_k$ must be finite (I'm looking for any, possibly loose, but finite bound)?



Note: I define the VC-dimension of a set of real-valued regressors as in 1 - Definition 2.3 in Shen et al. (2023). Namely, for any class $\mathcal{H}\subseteq \mathbb{R}^{\mathbb{R}^n}$ $$ \operatorname{VCdim}(\mathcal{H}):=\operatorname{VC-dim}(\{I_{(0,\infty)}\circ f:\,f\in \mathcal{H}\}) ,$$ where $\operatorname{VC-dim}(\{I_{(0,\infty)}\circ f:\,f\in \mathcal{H}\})$ denotes the standard VC dimension for classifiers.; so one considers all binary classifiers obtained by post-composing with the indicator function of the positive real line and considers the classical VC-dimension thereof.

Note, if $\mathcal{H}\subseteq \{0,1\}^{\mathbb{R}^n}$ then this coincides with the classical definition of VC-dimension.

  1. Shen et al. (2023) - Optimal approximation rate of ReLU networks in terms of width and depth
$\endgroup$
14
  • 1
    $\begingroup$ By $f =\sum_{i=1}^{k_0}\, f_n|_{C_n}$, do you actually mean $f= \sum_{n=1}^{k_0}\, f_n 1_{C_n}$ ?? $\endgroup$ Commented Jan 2 at 22:44
  • 1
    $\begingroup$ Also, what exactly do you mean by "a hypothesis class of maps" here? Are the values the maps in a set of cardinality $2$? If not, how do you define the VC dimension? $\endgroup$ Commented Jan 2 at 23:00
  • 1
    $\begingroup$ Also, by $\sum_{i=1}^{k_0}$ you probably mean $\sum_{n=1}^{k_0}$, right? $\endgroup$ Commented Jan 2 at 23:02
  • 1
    $\begingroup$ How then do you define the VC dimension? $\endgroup$ Commented Jan 3 at 0:21
  • 1
    $\begingroup$ What is the role of the $p_i$'s? $\endgroup$ Commented Jan 3 at 2:57

2 Answers 2

4
$\begingroup$

As noted by Iosif Pinelis, we can reformulate the problem in terms of classical set systems in the following way:

Let $\mathcal{H}$ be a set system on $\mathbb{R}^n$ with VC-dimension $d$. Fix $k\geq 1$ and define $\mathcal{H}_k$ to be the collection of all sets of the form

$$ \bigcup_{i=1}^{k_0}H_i\cap C_i $$ where $k_0\leq k$, each $H_i$ is in $\mathcal{H}$, and $(C_1,\ldots,C_{k_0})$ is the Voronoi partition of cells corresponding to some $p_1,\ldots,p_{k_0}$ in $\mathbb{R}^n$.

Theorem. $\mathcal{H}_k$ has finite VC-dimension bounded in terms of $k$, $d$, and $n$.

The proof relies on the following tools.

Fact. The set system of half spaces in $\mathbb{R}^n$ has VC-dimension $n+1$.

Lemma. Suppose $\mathcal{S}_1,\ldots,\mathcal{S}_m$ are set systems on some ground set $X$, and assume each $\mathcal{S}_i$ has VC-dimension at most $\ell$. Consider the set systems: $$ \mathcal{S}_{\cap}=\{S_1\cap\ldots\cap S_m:S_i\in\mathcal{S}_i\} $$ $$ \mathcal{S}_{\cup}=\{S_1\cup\ldots\cup S_m:S_i\in\mathcal{S}_i\} $$ Then $\mathcal{S}_{\cap}$ and $\mathcal{S}_{\cup}$ both have VC-dimension at most $2\ell m\log (3m)$.

The Fact is a standard exercise, and the Lemma is Lemma 3.2.3 in Learnability and the Vapnik-Chervonenkis Dimension by Blumer, Ehrenfeucht, Haussler, and Warmuth. The proof also is sketched in my answer here.

Proof of the Theorem. Let $\mathcal{C}$ be the collection consisting of the empty set and all subsets of $\mathbb{R}^n$ obtained as intersections of at most $k-1$ half spaces. Then every cell $C_i$ obtained as above is in $\mathcal{C}$. Moreover, $\mathcal{C}$ has VC-dimension at most $d_1=2(n+1)(k-1)\log(3k-3)$ by the Lemma and the Fact.

Next, let $\mathcal{H}'=\{H\cap C:H\in \mathcal{H},~C\in \mathcal{C}\}$. Applying the Lemma with $m=2$, $\mathcal{S}_1=\mathcal{H}$, and $\mathcal{S}_2=\mathcal{C}$, we conclude $\mathcal{H}'$ has VC-dimension at most $d_2=(4\log 6)\max\{d,d_1\}$.

Finally, let $\mathcal{H}''=\{H_1\cup\ldots\cup H_k:H_i\in \mathcal{H}'\}$. Then $\mathcal{H}_k\subseteq \mathcal{H}''$ and $\mathcal{H}''$ has VC-dimension at most $2d_2k\log(3k)$ by the Lemma.


Some additional remarks.

  1. The bound is certainly not tight since there is quite a bit of inefficiency in the proof; especially since $\mathcal{H}''$ is much bigger than $\mathcal{H}_k$. In particular, the proof is not actually using the partition feature of the $(C_1,\ldots,C_k)$ sequences (this is only used in checking that the Theorem accurately represents the original question about functions).

  2. In response to ABIM's comments, the bound cannot be improved to $kd$ in general. See Example 1 below, which also shows that the bound must depend on $n$.

  3. If one does not refine the original sequences $(\hat{C}_1,\ldots,\hat{C}_k)$ into partitions, then the Theorem is still true, but it does not accurately capture the original question. In fact, in this case the VC-dimension of the set systems from the question can be infinite, even when one just filters over a single sequence. See Example 2 below.

  4. Example 2 is not very satisfying since it is just exploiting the signal loss in the definition of VC-dimension for a class of real-valued functions from the cited reference (which I find a bit peculiar, but perhaps I'm not the right one to judge). There are different notions of VC-dimension for sets of functions which seem more likely to preserve finiteness in this situation (namely, not requiring the cells to partition and not even requiring the points to be pairwise distinct).

Example 1. Consider $\mathbb{R}^n$. Let $\mathcal{H}=\{\emptyset,\mathbb{R}^n\}$, which has VC-dimension $1$. Then $\mathcal{H}_k$ consists of all sets obtained as a union of some subset of a Veronoi sequence $(C_1,\ldots,C_{k_0})$ for some $k_0\leq k$. So, in particular, $\mathcal{H}_2$ is the set system of all half-spaces in $\mathcal{R}^n$ (together with $\emptyset$ and $\mathbb{R}^n$). By the Fact, $\mathcal{H}_2$ has VC-dimension $n+1$.

In the next example, I will let $\hat{\mathcal{H}}_k$ denote the set system of functions as defined in the original question, but using the non-partitioned sequences $(\hat{C}_1,\ldots,\hat{C}_k)$.

Example 2. Consider $\mathbb{R}^2$. Define $$ \mathcal{H}=\{I_S+1:S\subseteq\mathbb{R}^2\}\cup\{-I_{\mathbb{R}^2}\}. $$ Then VCdim$(\mathcal{H})=1$ since $I_{(0,\infty)}\circ f$ is either $I_{\mathbb{R}^2}$ or $I_\emptyset$ for any $f\in \mathcal{H}$. Now choose the sequence $(\hat{C}_1,\hat{C}_2)$ obtained from the points $(0,1)$ and $(0,-1)$. So $\hat{C}_1$ is the closed upper-half plane, $\hat{C}_2$ is the closed lower-half plane, and their intersection is the $x$-axis. Define $$ \mathcal{H}^*=\{fI_{\hat{C}_1}+gI_{\hat{C}_2}:f,g\in\mathcal{H}\}. $$ Then $\mathcal{H}^*\subseteq\hat{\mathcal{H}}_2$. We claim that $\mathcal{H}^*$ shatters the $x$-axis, which we call $A$ (as a subset of $\mathbb{R}^2$). To see this, fix $S\subseteq A$, and consider $g_S=(I_S+1)I_{\hat{C}_1}-I_{\mathbb{R}^2}I_{\hat{C}_2}\in\mathcal{H}^*$. Then $g_S$ simplifies to $I_{S\cup U}-I_{L}$ where $U$ is the open upper-half plane and $L$ is the open lower-half plane. So $$ (I_{(0,\infty)}\circ g_S)I_A=I_{S\cup U}I_A=I_S, $$ and we have cut out $S$ from $A$ using $\mathcal{H}^*$.

Note that $n\geq 2$ is necessary in the previous example since in dimension $1$ there are only $k-1$ intersection points in any given sequence (defined using pairwise distinct points), and hence the perturbation to the VC-dimension contributed by these points is finite. On the other hand, if one considers a more degenerate notion in which the $p_i$'s are not assumed to be pairwise distinct, then an example similar to the above can be built to obtain infinite VC-dimension (with a single sequence) even in dimension $1$.

$\endgroup$
5
  • $\begingroup$ A nit, I think OP and you are using different norms. Since the fact you mention assumes one is working with l1 balls. $\endgroup$
    – ABIM
    Commented Jan 5 at 4:25
  • $\begingroup$ @ABIM Maybe I'm confused, but I don't follow. The Fact isn't (fundamentally) about any norm, just about the set system of half-spaces in $\mathbb{R}^n$. A Voronoi cell (in the usual norm) is obtained by intersecting such regions. $\endgroup$ Commented Jan 5 at 13:29
  • $\begingroup$ @GabeConant Thanks for the increadibly clear answer, also you are correct on the tie-breaking (up to some possibly different permutation of induces, but no difference; I'll update the definition (it is as you assumed...I just thought it wouldn't matter, but it really does apparently :0 )). $\endgroup$ Commented Jan 5 at 14:35
  • $\begingroup$ @GabeConant Most importantly, so if I read your proof correctly since the upper bound is independent of the choice of distinct points $p_1,\dots,p_{k_0}$ ($k_0:=k$ in your case) then it holds for all such points; correct? $\endgroup$ Commented Jan 5 at 14:36
  • 1
    $\begingroup$ Saying "the upper bound is independent of the choice of points" sounds like saying "There is a bound $m$ such that for any fixed points, if I construct the set system using only the cell sequence for those fixed points, then the VC-dimension is at most $m$." This is true and one can take $m=kd$ by Iosef's answer (since there is only one partition here). However, I understood your question to mean that we do not fix the points, but instead construct one set system involving all cell partitions obtained from all choices of points. This is how $\mathcal{H}_k$ is defined in my answer. $\endgroup$ Commented Jan 5 at 14:43
7
$\begingroup$

$\newcommand\R{\mathbb R}\newcommand\vc{\mathrm{VC}}\renewcommand\H{\mathcal H}\newcommand\CC{\mathcal C}$As pointed out in Gabe Conant's comment, the following answer holds only in its original form, when then partition is fixed, that is, when the set $\CC$ of partitions of the set $X$ is a singleton.


The problem can restated (somewhat more generally), as follows:

Let $\H$ be a set of subsets of some set $X$. For a given natural $k$, let $\CC$ be a set of partitions $(C_1,\dots,C_k)$ of $X$ (some of the $C_j$'s may be empty). Let \begin{equation} \H_k:=\Big\{\bigcup_{j\in[k]}(H_j\cap C_j)\colon (C_1,\dots,C_k)\in\CC,\ H_j\in\H\ \forall j\in[k]\Big\}, \end{equation} where $[k]:=\{1,\dots,k\}$. Suppose that the VC dimension $\dim_\vc\H$ of $\H$ is $\le d$ for some natural $d$. Is then $\dim_\vc\H_k$ finite?

The answer to this is: Yes, $\dim_\vc\H_k\le kd$.

Indeed, the condition $\dim_\vc\H\le d$ means that all the sets $A$ shattered by $\H$ are of cardinality $|A|\le d$; a set $A\subseteq X$ is said to be shattered by $\H$ if the set $\H\cap A:=\{H\cap A\colon H\in\H\}$ coincides with the power set of $A$.

So, for any $A\subseteq X$, \begin{align*} &\H_k\text{ shatters }A \\ &\implies\exists (C_1,\dots,C_k)\in\CC\ \forall j\in[k]\ \H_k\cap C_j\text{ shatters }A\cap C_j \\ &\implies\exists (C_1,\dots,C_k)\in\CC\ \forall j\in[k]\ \H\text{ shatters }A\cap C_j \\ &\implies\exists (C_1,\dots,C_k)\in\CC\ \forall j\in[k]\ |A\cap C_j|\le d \\ &\implies|A|\le kd. \quad\Box \end{align*}

$\endgroup$
15
  • 1
    $\begingroup$ @Math_Newbie : Yes, the bound $|A|≤k_0d≤kd$ holds independently of the choice of a partition. No, one should not take the union over the set of all partitions of $X$ with at most $k$ parts. $\endgroup$ Commented Jan 3 at 3:37
  • 1
    $\begingroup$ @Math_Newbie : Do not use phrases such as "free parameters", with unclear meaning. Instead, use carefully placed quantifiers, $\forall$ and $\exists$. $\endgroup$ Commented Jan 3 at 3:44
  • 1
    $\begingroup$ @Math_Newbie : In such a case, just replace each of the three entries of "$\forall j\in[k]$" in the four-line display by "for some partition $(C_1,\dots,C_k)$ of $X$ and all $j\in[k]$". $\endgroup$ Commented Jan 3 at 4:01
  • 1
    $\begingroup$ However I believe the answer to the original question is still yes since the partitions involved appear to consist of convex polytopes with bounded complexity. But this depends on my understanding of the sets $C_i$, which are still not defined correctly in the original question. If OP corrects this definition, and it agrees with what I am guessing it is, then I can post an argument that the VC-dimension is finite. The bound would depend on $d$, $k$, and $n$, and would certainly not be optimal. $\endgroup$ Commented Jan 3 at 16:30
  • 1
    $\begingroup$ @ABIM Perhaps your conjecture about recovering $kd$ could work in $n=1$, but it fails in general. I will add an example to my answer. $\endgroup$ Commented Jan 5 at 15:21

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .