I'm learning SVM and its written everywhere that the minimal number of support vectors is 2? But I couldn't find any formal proofs of that. Why cant there be less than 2 support vectors? Can somebody give an intuitive way of understanding that and a formal explanation?
You have a training dataset of points $x_1,\dots,x_n$, say in $V:=\mathbb R^d$, each classified as green or blue (say). The dataset is linearly separable if there is a hyperplane $H$ in $V$ such that all the green data points are in one open half-space (say $H^+$) bounded by $H$ and all the blue data points are in the other open half-space (say $H^-$) bounded by $H$. Suppose that such a separating hyperplane $H$ indeed exists. (Otherwise, a standard remedy is to map (possibly nonlinearly) the dataset into another space, possibly of a higher dimension.) The separating hyperplane $H$ is called the maximum-margin hyperplane if the minimal distance $d(H):=\min_i d(x_i,H)$ from the dataset points $x_i$ to $H$ is maximal over all choices of separating hyperplanes; we are assuming here that the green and blue subsets of the dataset are each nonempty. See the picture in Section "Linear SVM". The points $x_i$ of the dataset that are the closest ones to the maximum-margin hyperplane $H$ are called the support vectors. Clearly, there is at least one such vector in each of the open half-spaces, $H^\pm$. (In the picture, the boundaries of the corresponding dots are highlighted as black.) So, there are at least two support vectors: at least one of them on either side of the maximum-margin hyperplane.