How to construct a linear regression model with an unbalanced sample design?

Question

A relatively simple question, but I cannot seem to find anything relevant to this. In fact, I'm not even sure of "unbalanced design" is even the right terminology here, but it was suggested to me.

The problem may be described as follows. Let's say you have a partition of your population into some groups - A, B and C. These groups contain 100, 200 and 500 individuals respectively. You don't know that - you only know the relative sizes (so, B is 2x the size of A, C is 5x the size of A, etc.). Furthermore, you have three random samples, of sizes 50, 20 and 100 from each of these.

How does one construct a linear regression here (nothing fancy) that combines the data from all three samples?

Answer 1

I think that you should use a weighted regression here. Suppose that you have $n$ data points, for each you see the predictor vector $\vec{x_i}$ and response variable $y_i$. Suppose moreover that these points belong to $m$ different sub-populations, with proportions $\alpha_1,..,\alpha_m$ summing to one (in your case you have 3 sub-populations with proportions $1/8, 2/8, 5/8$) and observed sample sizes $n_1, .., n_m$ summing to $n$ (in your case $50, 20, 100$). Then instead of performing ordinary least-square regression, i.e. finding $\vec{\beta}$ minimizing the sum of squares: $\sum_i (y_i - \vec{x_i} \vec{\beta})^2$, you should minimize: $\sum_i w_i (y_i - \vec{x_i} \vec{\beta})^2$. The weights $w_i$ are determined as follows: If the $i$-th observation was taken from the $j$-th sup-population, then set $w_i = \alpha_j / n_j$. This guarantees that the total weight in the regression cost function of each sub-population will be proportional to it's fraction in the population. For example, in your case each observation from $A$ should be assigned a weight $\frac{1/8}{50}$.

Answer 2

The class imbalance you describe isn't that problematic. Just use regression as you would normally do. In cases where the imbalance ratio is around 1:10000 you would need to use different methods to deal with the problem such as: Downsampling, upsampling, SMOTE, etc.

Take a look at: http://appliedpredictivemodeling.com/ There is a complete chapter regarding the problem. But as I say, in your case it's not needed.