1
$\begingroup$

I have a large linear regression where all the independent variables are logical (ie TRUE/FALSE) and sparse. The data has roughly 10,000 variables and 10 million observations, on average around 20 true values per observation. I’d like to compress the variables or group them together to make the system less unwieldy. For instance, I can combine several variables by logical OR into a new grouped variable. Then I can run a new linear regression on the grouped variables.

I have one grouping based on the interpretation of the variables, and the $R^2$ of its regression is much better than random but also clearly not optimal. My goal is to group the 10.000 original variables into roughly 100 grouped variables with roughly 100 of the original variables in each.

Question: How do I decide which variables to group so that the loss of predictive power (measured as $R^2$ of the models) is minimal? Or how can I improve a grouping?

The number of ways to group the variables is finite, but too big to search through, so I'm looking for an algorithm that is significantly better than random. Even the number of ways to alter a grouping by moving a single variable into another group is too big to try all of them.

Edit in response to comment: the dependent variable is numeric.

Edit in response to PCA: The data is of medical origin, the independent variables are diagnoses, the observations are patients and the dependent variable is treatment cost (in some part of the health industry). In particular, the dependent variable is always positive and the vast majority of model coefficients in the big model is positive as well. Hence using only a subset of the variables would produce a significiant bias for underestimating the dependent variable.

Improve this question
$\endgroup$
3
  • 4
    $\begingroup$ "Compressing data" is a task that is typically done using SVD / principal component analysis / latent semantic analysis (same idea, just different names for it from various fields). Are you familiar with this approach? It does not do exactly what you ask here (it does not identify uniquely single variables that you can keep and remove, but rather finds meaningful linear combinations of them), but it's a standard algorithm and you should definitely start from there. $\endgroup$
    – Federico Poloni
    Commented Apr 4, 2019 at 12:23
  • $\begingroup$ @FedericoPoloni Principal component analysis would rank my independent variables by order of importance. Presumably a model that only uses the 100 most important independent variables would be not that much worse than the full 10000 variable model. However, due to the sparsity I would expect it to be a worse than a reasonably good model with grouped variables. $\endgroup$
    – quarague
    Commented Apr 4, 2019 at 12:59
  • 1
    $\begingroup$ Another search term is "dimension reduction" $\endgroup$
    – Neal
    Commented Apr 4, 2019 at 12:59

0

Reset to default

You must log in to answer this question.