Return to Question

Let $\{\mathbf{x}_i, y_i \}$ be a set of binary-labeled samples ($\mathbf{x}_i \in \mathbb{R}^d, y_i \in \{a,b\}, a,b\in\mathbb{R}$). Let $\{ \mathbf{x}'_i, y_i \}$ be also such a set. Define $\mathbf{X} = [\mathbf{x}_1 \dots \mathbf{x}_N]^T , \mathbf{y}=[y_1 \dots y_N]^T$ and similarly for $\mathbf{X}'$. Define $\beta = (\mathbf{X}^T \mathbf{X})^{-1}\mathbf{X}^T \mathbf{y}$ and $\beta' = (\mathbf{X}'^T \mathbf{X}')^{-1}\mathbf{X}'^T \mathbf{y}$. Define $\widehat{\mathbf{y}} = \mathbf{X} \beta$, $\pmb{\epsilon} = \widehat{\mathbf{y}} - \mathbf{y}$, and similarly for $\widehat{\mathbf{y}'}, \pmb{\epsilon}'$.

Then how can we prove the following conjecture?

Conjcture. (a) Assume for any $i,j,k$ with $y_i = y_j \neq y_k $, $$ ||\mathbf{x}_i - \mathbf{x}_j || = ||\mathbf{x}'_{i} - \mathbf{x}'_{j}||, \qquad || \mathbf{x}_i - \mathbf{x}_k || < || \mathbf{x}'_{i} - \mathbf{x}'_{k} || $$ Then $||\pmb{\epsilon}|| > ||\pmb{\epsilon}'||$.

Probability perspective. Now we take a probabilistic perspective. Let $N$ be a natural number. Let $x$ be a random vector such that the above $\mathbf{x}_i$ are the realizations of it. Also define similarly $x'$. Let $y$ be a random vector that models $y_i$. (I.e., $y = \text{label}(x) = \text{label}(x')$.) Define $\widehat{y} = x^T\beta$ and $\widehat{y'} = x'^T\beta'$ where $$ \beta = (\mathbf{X}_n^T\mathbf{X}_N)^{-1}\mathbf{X}_N^T\mathbf{y}_N $$ with $X_n = [\mathbf{x}_1 \cdots \mathbf{x}_N]^T$ for the realizations $\mathbf{x}_i$ of $x$ and $\mathbf{y}_N = [y_1 \dots y_N]^T$ ($y_i = \text{label}(\mathbf{x}_i)$), and $\beta'$ is defined similarly. Define $\epsilon = \widehat{y} - y$ and similarly for $\epsilon'$.

Conjecture. (b). Assume $||\mathbb{E}[x|y=0] - \mathbb{E}[x|y=1]|| < ||\mathbb{E}[x'|y=0] - \mathbb{E}[x'|y=1]|| $ while $ \text{var}(x|y=i) =\text{var}(x'|y=i)$ for $i=0,1$. Then $$ \text{var}(\epsilon) > \text{var}(\epsilon'). $$

(c) The above holds true when $N \to \infty$.

Question. How can I possibly approach this type of problem? What kind of techniques should I consider? Should I ease or changethe problem?

Basically what the conjecture says is that if we increase inter-class distances while preserving intra-class variances, the regression error decreases. Intuitively I think this is quite clear especially for simple linear regression with one feature variable.

Let $\{\mathbf{x}_i, y_i \}$ be a set of binary-labeled samples ($\mathbf{x}_i \in \mathbb{R}^d, y_i \in \{a,b\}, a,b\in\mathbb{R}$). Let $\{ \mathbf{x}'_i, y_i \}$ be also such a set. Define $\mathbf{X} = [\mathbf{x}_1 \dots \mathbf{x}_N]^T , \mathbf{y}=[y_1 \dots y_N]^T$ and similarly for $\mathbf{X}'$. Define $\beta = (\mathbf{X}^T \mathbf{X})^{-1}\mathbf{X}^T \mathbf{y}$ and $\beta' = (\mathbf{X}'^T \mathbf{X}')^{-1}\mathbf{X}'^T \mathbf{y}$. Define $\widehat{\mathbf{y}} = \mathbf{X} \beta$, $\pmb{\varepsilon} = \widehat{\mathbf{y}} - \mathbf{y}$, and similarly for $\widehat{\mathbf{y}'}, \pmb{\varepsilon}'$.

Then how can we prove the following conjecture?

Conjcture. (a) Assume for any $i,j,k$ with $y_i = y_j \neq y_k $, $$ \|\mathbf{x}_i - \mathbf{x}_j \| = \|\mathbf{x}'_{i} - \mathbf{x}'_{j}\|, \qquad \| \mathbf{x}_i - \mathbf{x}_k \| < \| \mathbf{x}'_{i} - \mathbf{x}'_{k} \| $$ Then $\|\pmb{\varepsilon}\| > \|\pmb{\varepsilon}'\|$.

Probability perspective. Now we take a probabilistic perspective. Let $N$ be a natural number. Let $x$ be a random vector such that the above $\mathbf{x}_i$ are the realizations of it. Also define similarly $x'$. Let $y$ be a random vector that models $y_i$. (I.e., $y = \operatorname{label}(x) = \operatorname{label}(x')$.) Define $\widehat{y} = x^T\beta$ and $\widehat{y'} = x'^T\beta'$ where $$ \beta = (\mathbf{X}_n^T\mathbf{X}_N)^{-1}\mathbf{X}_N^T\mathbf{y}_N $$ with $X_n = [\mathbf{x}_1 \cdots \mathbf{x}_N]^T$ for the realizations $\mathbf{x}_i$ of $x$ and $\mathbf{y}_N = [y_1 \dots y_N]^T$ ($y_i = \operatorname{label}(\mathbf{x}_i)$), and $\beta'$ is defined similarly. Define $\varepsilon = \widehat{y} - y$ and similarly for $\varepsilon'$.

Conjecture. (b). Assume $\|\mathbb{E}[x\mid y=0] - \mathbb{E}[x|y=1]\| < \|\mathbb{E}[x'\mid y=0] - \mathbb{E}[x'\mid y=1]\| $ while $ \operatorname{var}(x\mid y=i) =\operatorname{var}(x'\mid y=i)$ for $i=0,1$. Then $$ \operatorname{var}(\varepsilon) > \operatorname{var}(\varepsilon'). $$

(c) The above holds true when $N \to \infty$.

Question. How can I possibly approach this type of problem? What kind of techniques should I consider? Should I ease or changethe problem?

Basically what the conjecture says is that if we increase inter-class distances while preserving intra-class variances, the regression error decreases. Intuitively I think this is quite clear especially for simple linear regression with one feature variable.

Let $\{\mathbf{x}_i, y_i \}$ be a set of binary-labeled samples ($\mathbf{x}_i \in \mathbb{R}^d, y_i \in \{a,b\}, a,b\in\mathbb{R}$). Let $\{ \mathbf{x}'_i, y_i \}$ be also such a set. Define $\mathbf{X} = [\mathbf{x}_1 \dots \mathbf{x}_N]^T , \mathbf{y}=[y_1 \dots y_N]^T$ and similarly for $\mathbf{X}'$. Define $\beta = (\mathbf{X}^T \mathbf{X})^{-1}\mathbf{X}^T \mathbf{y}$ and $\beta' = (\mathbf{X}'^T \mathbf{X}')^{-1}\mathbf{X}'^T \mathbf{y}$. Define $\widehat{\mathbf{y}} = \mathbf{X} \beta$, $\pmb{\epsilon} = \widehat{\mathbf{y}} - \mathbf{y}$, and similarly for $\widehat{\mathbf{y}'}, \pmb{\epsilon}'$.

Then how can we prove the following conjecture?

Conjcture. (a) Assume for any $i,j,k,i',j',k'$ with $y_i = y_{i'} = y_j = y_{j'} \neq y_k = y_{k'}$, $$ ||\mathbf{x}_i - \mathbf{x}_j || = ||\mathbf{x}'_{i'} - \mathbf{x}'_{j'}||, \qquad || \mathbf{x}_i - \mathbf{x}_k || < || \mathbf{x}'_{i'} - \mathbf{x}'_{k'} || $$ Then $||\pmb{\epsilon}|| > ||\pmb{\epsilon}'||$.

Probability perspective. Now we take a probabilistic perspective. Let $N$ be a natural number. Let $x$ be a random vector such that the above $\mathbf{x}_i$ are the realizations of it. Also define similarly $x'$. Let $y$ be a random vector that models $y_i$. (I.e., $y = \text{label}(x) = \text{label}(x')$.) Define $\widehat{y} = x^T\beta$ and $\widehat{y'} = x'^T\beta'$ where $$ \beta = (\mathbf{X}_n^T\mathbf{X}_N)^{-1}\mathbf{X}_N^T\mathbf{y}_N $$ with $X_n = [\mathbf{x}_1 \cdots \mathbf{x}_N]^T$ for the realizations $\mathbf{x}_i$ of $x$ and $\mathbf{y}_N = [y_1 \dots y_N]^T$ ($y_i = \text{label}(\mathbf{x}_i)$), and $\beta'$ is defined similarly. Define $\epsilon = \widehat{y} - y$ and similarly for $\epsilon'$.

Conjecture. (b). Assume $||\mathbb{E}[x|y=0] - \mathbb{E}[x|y=1]|| < ||\mathbb{E}[x'|y=0] - \mathbb{E}[x'|y=1]|| $ while $ \text{var}(x|y=i) =\text{var}(x'|y=i)$ for $i=0,1$. Then $$ \text{var}(\epsilon) > \text{var}(\epsilon'). $$

(c) The above holds true when $N \to \infty$.

Question. How can I possibly approach this type of problem? What kind of techniques should I consider? Should I ease or changethe problem?

Basically what the conjecture says is that if we increase inter-class distances while preserving intra-class variances, the regression error decreases. Intuitively I think this is quite clear especially for simple linear regression with one feature variable.

Let $\{\mathbf{x}_i, y_i \}$ be a set of binary-labeled samples ($\mathbf{x}_i \in \mathbb{R}^d, y_i \in \{a,b\}, a,b\in\mathbb{R}$). Let $\{ \mathbf{x}'_i, y_i \}$ be also such a set. Define $\mathbf{X} = [\mathbf{x}_1 \dots \mathbf{x}_N]^T , \mathbf{y}=[y_1 \dots y_N]^T$ and similarly for $\mathbf{X}'$. Define $\beta = (\mathbf{X}^T \mathbf{X})^{-1}\mathbf{X}^T \mathbf{y}$ and $\beta' = (\mathbf{X}'^T \mathbf{X}')^{-1}\mathbf{X}'^T \mathbf{y}$. Define $\widehat{\mathbf{y}} = \mathbf{X} \beta$, $\pmb{\epsilon} = \widehat{\mathbf{y}} - \mathbf{y}$, and similarly for $\widehat{\mathbf{y}'}, \pmb{\epsilon}'$.

Then how can we prove the following conjecture?

Conjcture. (a) Assume for any $i,j,k$ with $y_i = y_j \neq y_k $, $$ ||\mathbf{x}_i - \mathbf{x}_j || = ||\mathbf{x}'_{i} - \mathbf{x}'_{j}||, \qquad || \mathbf{x}_i - \mathbf{x}_k || < || \mathbf{x}'_{i} - \mathbf{x}'_{k} || $$ Then $||\pmb{\epsilon}|| > ||\pmb{\epsilon}'||$.

Probability perspective. Now we take a probabilistic perspective. Let $N$ be a natural number. Let $x$ be a random vector such that the above $\mathbf{x}_i$ are the realizations of it. Also define similarly $x'$. Let $y$ be a random vector that models $y_i$. (I.e., $y = \text{label}(x) = \text{label}(x')$.) Define $\widehat{y} = x^T\beta$ and $\widehat{y'} = x'^T\beta'$ where $$ \beta = (\mathbf{X}_n^T\mathbf{X}_N)^{-1}\mathbf{X}_N^T\mathbf{y}_N $$ with $X_n = [\mathbf{x}_1 \cdots \mathbf{x}_N]^T$ for the realizations $\mathbf{x}_i$ of $x$ and $\mathbf{y}_N = [y_1 \dots y_N]^T$ ($y_i = \text{label}(\mathbf{x}_i)$), and $\beta'$ is defined similarly. Define $\epsilon = \widehat{y} - y$ and similarly for $\epsilon'$.

Conjecture. (b). Assume $||\mathbb{E}[x|y=0] - \mathbb{E}[x|y=1]|| < ||\mathbb{E}[x'|y=0] - \mathbb{E}[x'|y=1]|| $ while $ \text{var}(x|y=i) =\text{var}(x'|y=i)$ for $i=0,1$. Then $$ \text{var}(\epsilon) > \text{var}(\epsilon'). $$

(c) The above holds true when $N \to \infty$.

Question. How can I possibly approach this type of problem? What kind of techniques should I consider? Should I ease or changethe problem?

Basically what the conjecture says is that if we increase inter-class distances while preserving intra-class variances, the regression error decreases. Intuitively I think this is quite clear especially for simple linear regression with one feature variable.

Let $\{\mathbf{x}_i, y_i \}$ be a set of binary-labeled samples ($\mathbf{x}_i \in \mathbb{R}^d, y_i \in \{a,b\}, a,b\in\mathbb{R}$). Let $\{ \mathbf{x}'_i, y_i \}$ be also such a set. Define $\mathbf{X} = [\mathbf{x}_1 \dots \mathbf{x}_N]^T , \mathbf{y}=[y_1 \dots y_N]^T$ and similarly for $\mathbf{X}'$. Define $\beta = (\mathbf{X}^T \mathbf{X})^{-1}\mathbf{X}^T \mathbf{y}$ and $\beta' = (\mathbf{X}'^T \mathbf{X}')^{-1}\mathbf{X}'^T \mathbf{y}$. Define $\widehat{\mathbf{y}} = \mathbf{X} \beta$, $\pmb{\epsilon} = \widehat{\mathbf{y}} - \mathbf{y}$, and similarly for $\widehat{\mathbf{y}'}, \pmb{\epsilon}'$.

Then how can we prove the following conjecture?

Conjcture. (a) Assume for any $i,j,k,i',j',k'$ with $y_i = y_{i'} = y_j = y_{j'} \neq y_k = y_{k'}$, $$ ||\mathbf{x}_i - \mathbf{x}_j || = ||\mathbf{x}'_{i'} - \mathbf{x}'_{j'}||, \qquad || \mathbf{x}_i - \mathbf{x}_k || < || \mathbf{x}'_{i'} - \mathbf{x}'_{k'} || $$ Then $||\pmb{\epsilon}|| > ||\pmb{\epsilon}'||$.

Probability perspective. Now we take a probabilistic perspective. Let $N$ be a natural number. Let $x$ be a random vector such that the above $\mathbf{x}_i$ are the realizations of it. Also define similarly $x'$. Let $y$ be a random vector that models $y_i$. (I.e., $y = \text{label}(x) = \text{label}(x')$.) Define $\widehat{y} = x^T\beta$ and $\widehat{y'} = x'^T\beta'$ where $$ \beta = (\mathbf{X}_n^T\mathbf{X}_N)^{-1}\mathbf{X}_N^T\mathbf{y}_N $$ with $X_n = [\mathbf{x}_1 \cdots \mathbf{x}_N]^T$ for the realizations $\mathbf{x}_i$ of $x$ and $\mathbf{y}_N = [y_1 \dots y_N]^T$ ($y_i = \text{label}(\mathbf{x}_i)$), and $\beta'$ is defined similarly. Define $\epsilon = \widehat{y} - y$ and similarly for $\epsilon'$.

Conjecture. (b). Assume $||\mathbb{E}[x|y=0] - \mathbb{E}[x|y=1]|| < ||\mathbb{E}[x'|y=0] - \mathbb{E}[x'|y=1]|| $ while $ \text{var}(x|y=i) =\text{var}(x'|y=i)$ for $i=0,1$. Then $$ \text{var}(\epsilon) > \text{var}(\epsilon'). $$

(c) The above holds true when $N \to \infty$.

Question. How can I possibly approach this type of problem? What kind of techniques should I consider? Should I ease or changethe problem?

Basically what the conjecture says is that if we increase inter-class distances while preserving intra-class variances, the regression error decreases. Intuitively I think this is quite clear especially simple linear regression with one feature variable.

Let $\{\mathbf{x}_i, y_i \}$ be a set of binary-labeled samples ($\mathbf{x}_i \in \mathbb{R}^d, y_i \in \{a,b\}, a,b\in\mathbb{R}$). Let $\{ \mathbf{x}'_i, y_i \}$ be also such a set. Define $\mathbf{X} = [\mathbf{x}_1 \dots \mathbf{x}_N]^T , \mathbf{y}=[y_1 \dots y_N]^T$ and similarly for $\mathbf{X}'$. Define $\beta = (\mathbf{X}^T \mathbf{X})^{-1}\mathbf{X}^T \mathbf{y}$ and $\beta' = (\mathbf{X}'^T \mathbf{X}')^{-1}\mathbf{X}'^T \mathbf{y}$. Define $\widehat{\mathbf{y}} = \mathbf{X} \beta$, $\pmb{\epsilon} = \widehat{\mathbf{y}} - \mathbf{y}$, and similarly for $\widehat{\mathbf{y}'}, \pmb{\epsilon}'$.

Then how can we prove the following conjecture?

Conjcture. (a) Assume for any $i,j,k,i',j',k'$ with $y_i = y_{i'} = y_j = y_{j'} \neq y_k = y_{k'}$, $$ ||\mathbf{x}_i - \mathbf{x}_j || = ||\mathbf{x}'_{i'} - \mathbf{x}'_{j'}||, \qquad || \mathbf{x}_i - \mathbf{x}_k || < || \mathbf{x}'_{i'} - \mathbf{x}'_{k'} || $$ Then $||\pmb{\epsilon}|| > ||\pmb{\epsilon}'||$.

Probability perspective. Now we take a probabilistic perspective. Let $N$ be a natural number. Let $x$ be a random vector such that the above $\mathbf{x}_i$ are the realizations of it. Also define similarly $x'$. Let $y$ be a random vector that models $y_i$. (I.e., $y = \text{label}(x) = \text{label}(x')$.) Define $\widehat{y} = x^T\beta$ and $\widehat{y'} = x'^T\beta'$ where $$ \beta = (\mathbf{X}_n^T\mathbf{X}_N)^{-1}\mathbf{X}_N^T\mathbf{y}_N $$ with $X_n = [\mathbf{x}_1 \cdots \mathbf{x}_N]^T$ for the realizations $\mathbf{x}_i$ of $x$ and $\mathbf{y}_N = [y_1 \dots y_N]^T$ ($y_i = \text{label}(\mathbf{x}_i)$), and $\beta'$ is defined similarly. Define $\epsilon = \widehat{y} - y$ and similarly for $\epsilon'$.

Conjecture. (b). Assume $||\mathbb{E}[x|y=0] - \mathbb{E}[x|y=1]|| < ||\mathbb{E}[x'|y=0] - \mathbb{E}[x'|y=1]|| $ while $ \text{var}(x|y=i) =\text{var}(x'|y=i)$ for $i=0,1$. Then $$ \text{var}(\epsilon) > \text{var}(\epsilon'). $$

(c) The above holds true when $N \to \infty$.

Question. How can I possibly approach this type of problem? What kind of techniques should I consider? Should I ease or changethe problem?

Basically what the conjecture says is that if we increase inter-class distances while preserving intra-class variances, the regression error decreases. Intuitively I think this is quite clear especially for simple linear regression with one feature variable.