Let $\{\mathbf{x}_i, y_i \}$ be a set of binary-labeled samples ($\mathbf{x}_i \in \mathbb{R}^d, y_i \in \{a,b\}, a,b\in\mathbb{R}$). Let $\{ \mathbf{x}'_i, y_i \}$ be also such a set. Define $\mathbf{X} = [\mathbf{x}_1 \dots \mathbf{x}_N]^T , \mathbf{y}=[y_1 \dots y_N]^T$ and similarly for $\mathbf{X}'$. Define $\beta = (\mathbf{X}^T \mathbf{X})^{-1}\mathbf{X}^T \mathbf{y}$ and $\beta' = (\mathbf{X}'^T \mathbf{X}')^{-1}\mathbf{X}'^T \mathbf{y}$. Define $\widehat{\mathbf{y}} = \mathbf{X} \beta$, $\pmb{\epsilon} = \widehat{\mathbf{y}} - \mathbf{y}$, and similarly for $\widehat{\mathbf{y}'}, \pmb{\epsilon}'$.

Then how can we prove the following conjecture?

Conjcture. (a) Assume for any $i,j,k,i',j',k'$ with $y_i = y_{i'} = y_j = y_{j'} \neq y_k = y_{k'}$, $$ ||\mathbf{x}_i - \mathbf{x}_j || = ||\mathbf{x}'_{i'} - \mathbf{x}'_{j'}||, \qquad || \mathbf{x}_i - \mathbf{x}_k || < || \mathbf{x}'_{i'} - \mathbf{x}'_{k'} || $$ Then $||\pmb{\epsilon}|| > ||\pmb{\epsilon}'||$.

Probability perspective. Now we take a probabilistic perspective. Let $N$ be a natural number. Let $x$ be a random vector such that the above $\mathbf{x}_i$ are the realizations of it. Also define similarly $x'$. Let $y$ be a random vector that models $y_i$. (I.e., $y = \text{label}(x) = \text{label}(x')$.) Define $\widehat{y} = x^T\beta$ and $\widehat{y'} = x'^T\beta'$ where $$ \beta = (\mathbf{X}_n^T\mathbf{X}_N)^{-1}\mathbf{X}_N^T\mathbf{y}_N $$ with $X_n = [\mathbf{x}_1 \cdots \mathbf{x}_N]^T$ for the realizations $\mathbf{x}_i$ of $x$ and $\mathbf{y}_N = [y_1 \dots y_N]^T$ ($y_i = \text{label}(\mathbf{x}_i)$), and $\beta'$ is defined similarly. Define $\epsilon = \widehat{y} - y$ and similarly for $\epsilon'$.

Conjecture. (b). Assume $||\mathbb{E}[x|y=0] - \mathbb{E}[x|y=1]|| < ||\mathbb{E}[x'|y=0] - \mathbb{E}[x'|y=1]|| $ while $ \text{var}(x|y=i) =\text{var}(x'|y=i)$ for $i=0,1$. Then $$ \text{var}(\epsilon) > \text{var}(\epsilon'). $$

(c) The above holds true when $N \to \infty$.

Question. How can I possibly approach this type of problem? What kind of techniques should I consider? Should I ease or changethe problem?

Basically what the conjecture says is that if we increase inter-class distances while preserving intra-class variances, the regression error decreases. Intuitively I think this is quite clear especially simple linear regression with one feature variable.