Take the 2-minute tour ×
MathOverflow is a question and answer site for professional mathematicians. It's 100% free, no registration required.

I have a large data set, A, containing 100 x/y pairs. I've divided it into two smaller data sets, B and C, containing 30 and 70 x/y pairs respectively.

I have Pearson's product-moment correlation r for each of the two smaller data sets, B and C. Can I combine the correlation coefficients from the two smaller sets to generate the correlation coefficient for A?

This is for a programming problem I'm working on, and my dataset, A, is very large. I need to somehow calculate the correlation coefficient for it, but I'd like to split the dataset up into many smaller datasets, calculate the correlation for each small dataset, and then combine those correlations to get my result for the dataset as a whole. Is it possible?

Thanks!

share|improve this question

3 Answers 3

up vote 5 down vote accepted

You can't do that, as Gerry Myerson has pointed out.

If you want a way to break down the computation, though, go back to one of the formulas for it:

$$ r_{xy} = {n \sum_i x_i y_i - \sum_i x_i \sum_i y_i \over \sqrt{n \sum_i x_i^2 - (\sum x_i)^2} \sqrt{n \sum_i y_i^2 - (\sum_i y_i)^2}}. $$

(See the wikipedia article, under "mathematical properties".)

So you just need to know $n, \sum_i x_i y_i, \sum_i x_i$ and $\sum_i y_i$ for the whole data set. And these will just be the sum of the corresponding quantities for each subset.

share|improve this answer
    
Good, and more helpful than my answer. But your displayed equation is not typesetting. Is there something wrong with your TeX, or is the problem at my end? –  Gerry Myerson Mar 9 '11 at 12:11
    
Gerry: thanks! The original question is actually one I got (in slightly different form) from a student a couple weeks ago. Douglas: thanks for fixing the TeX when I hadn't even woken up yet. –  Michael Lugo Mar 9 '11 at 18:30
    
Wouldn't I also need the sum of x squared (and y squared)? Thanks, this is probably the solution I'm going to use now. –  Imbue Mar 10 '11 at 7:07
    
OK, I just implemented your solution, and it appears to work great. Do you know what Wikipedia means, exactly, when they say this formula may be numerically unstable? Thanks again! –  Imbue Mar 10 '11 at 7:24
    
Imbue: to answer your first comment, you're right that you need to know $\sum (x_i^2)$ and $\sum (y_i^2)$, as you probably figured out while you were implementing this. The issue of numerical instability probably occurs because you're taking differences between what might be large numbers that are close together. John Cook has written about this in more detail: johndcook.com/blog/2008/11/05/… –  Michael Lugo Mar 10 '11 at 16:19

Seems unlikely. Suppose every data point in $B$ has $y=2x$, while every data point in $C$ has $y=x/2$. Then $B$ has terrific correlation, and so does $C$, but $A$ doesn't.

share|improve this answer
    
Thanks! I'm pretty new to this stuff (statistics), and I had a suspicion that it couldn't be done, but your example was easy to follow and spot on. Great explanation! Now I'll just have to find a way around it somehow. –  Imbue Mar 9 '11 at 4:34

In addition to what Gerry had already mentioned: of course, if the subsamples B and C were true random samples from the full sample, call it "population", meaning they are "representative" then the correlation-coefficients of the smaller samples are always estimators for that of the "population", and if you use two or more random subsamples the estimated population-coefficient is somehow an average.

But well, as you state your problem, it looks very likely to me that B and C are not such random-samples but are taken using some criterion. If such a criterion is existent then one should determine whether it distorts the randomness of the subsamples: if you take,for instance, B from the left edge of the whole data-cloud in a scatterplot and C from the right edge then the best-fit-lines in that subsamples may have completely different slopes and variances around them.

[update2] If such an averaging of correlations is actually meaningful in your problem (your subsamples are random and not too small) then I'd recommend to average the z-transforms of the correlation-coefficients. That means $$ r_{\text{est}} = \tanh(\frac{\sum_{k=1}^{s}\tanh^{-1}(r_k)}{s}) $$ where $s$ is the number of samples, because that fisher-transformation approximates by conversion a correlation-coefficient into a z-variable (normal distributed, mean=0, infinite range) where the averaging over the arithmetical mean is more meaningful.

[update]
Here I show examples where the subsamples were taken randomly. I generated correlated data of a population with n= 2000, normal distributed with mean=0, stddev=1, correlation r~ 0.35 . I show the variation of the occuring correlations if random samples of n=20, n=50, n=100 are drawn. For each sample-size I did 500 experiments and documented the frequencies of occuring correlations r in steps of about 0.1.

sample-n:   20          avg r:      0.37760   experiments: 500 
pop-n   :   2000        pop r:      0.35247

  low r       high r    freq    
--------------------------------    
-0.2023     -0.2023      1
-0.1807     -0.0948      8
-0.0878      0.0101      15
 0.0205      0.1068      25
 0.1112      0.2101      60
 0.2123      0.3098      100
 0.3113      0.4073      81
 0.4109      0.5102      83
 0.5109      0.6100      73
 0.6107      0.7078      44
 0.7122      0.7891      10
===================================

sample-n:   50          avg r:      0.36040
pop-n   :   2000        pop r:      0.35247

  low r       high r    freq    
--------------------------------
-0.1011     -0.1011      1
 0.0175      0.1027      9
 0.1098      0.2022      55
 0.2056      0.3027      108
 0.3043      0.4027      150
 0.4047      0.5030      124
 0.5045      0.6024      45
 0.6099      0.6982      8
===================================


sample-n:   100         avg r:      0.35657
pop-n   :   2000        pop r:      0.35247

  low r       high r    freq    
----------------------------------
 0.0504      0.0703      3
 0.1139      0.2032      20
 0.2054      0.3034      115
 0.3055      0.4038      217
 0.4047      0.4956      133
 0.5046      0.5471      12
===================================

One can determine confidence-intervals for the correlations; that intervals narrow with increasing size of the samples.
But this all is only useful if the different samples are really random and not taken by some systematic criterion.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.