Combining Correlation Coefficients - MathOverflow most recent 30 from http://mathoverflow.net 2013-05-23T05:24:57Z http://mathoverflow.net/feeds/question/57908 http://www.creativecommons.org/licenses/by-nc/2.5/rdf http://mathoverflow.net/questions/57908/combining-correlation-coefficients Combining Correlation Coefficients Imbue 2011-03-09T04:21:44Z 2011-03-11T22:05:43Z <p>I have a large data set, A, containing 100 x/y pairs. I've divided it into two smaller data sets, B and C, containing 30 and 70 x/y pairs respectively.</p> <p>I have <a href="http://en.wikipedia.org/wiki/Pearson_product-moment_correlation_coefficient" rel="nofollow">Pearson's product-moment correlation r</a> for each of the two smaller data sets, B and C. Can I combine the correlation coefficients from the two smaller sets to generate the correlation coefficient for A?</p> <p>This is for a programming problem I'm working on, and my dataset, A, is very large. I need to somehow calculate the correlation coefficient for it, but I'd like to split the dataset up into many smaller datasets, calculate the correlation for each small dataset, and then combine those correlations to get my result for the dataset as a whole. Is it possible?</p> <p>Thanks!</p> http://mathoverflow.net/questions/57908/combining-correlation-coefficients/57912#57912 Answer by Gerry Myerson for Combining Correlation Coefficients Gerry Myerson 2011-03-09T04:31:39Z 2011-03-09T04:31:39Z <p>Seems unlikely. Suppose every data point in $B$ has $y=2x$, while every data point in $C$ has $y=x/2$. Then $B$ has terrific correlation, and so does $C$, but $A$ doesn't. </p> http://mathoverflow.net/questions/57908/combining-correlation-coefficients/57914#57914 Answer by Michael Lugo for Combining Correlation Coefficients Michael Lugo 2011-03-09T05:00:23Z 2011-03-09T14:45:16Z <p>You can't do that, as Gerry Myerson has pointed out.</p> <p>If you want a way to break down the computation, though, go back to one of the formulas for it:</p> <p>$$r_{xy} = {n \sum_i x_i y_i - \sum_i x_i \sum_i y_i \over \sqrt{n \sum_i x_i^2 - (\sum x_i)^2} \sqrt{n \sum_i y_i^2 - (\sum_i y_i)^2}}.$$</p> <p>(See the <a href="http://en.wikipedia.org/wiki/Pearson_product-moment_correlation_coefficient" rel="nofollow">wikipedia article</a>, under "mathematical properties".) </p> <p>So you just need to know $n, \sum_i x_i y_i, \sum_i x_i$ and $\sum_i y_i$ for the whole data set. And these will just be the sum of the corresponding quantities for each subset.</p> http://mathoverflow.net/questions/57908/combining-correlation-coefficients/57921#57921 Answer by Gottfried Helms for Combining Correlation Coefficients Gottfried Helms 2011-03-09T06:28:00Z 2011-03-11T20:41:23Z <p>In addition to what Gerry had already mentioned: of course, if the subsamples <strong>B</strong> and <strong>C</strong> were true random samples from the full sample, call it "population", meaning they are "representative" then the correlation-coefficients of the smaller samples are always estimators for that of the "population", and if you use two or more random subsamples the estimated population-coefficient is somehow an average. </p> <p>But well, as you state your problem, it looks very likely to me that <strong>B</strong> and <strong>C</strong> are not such random-samples but are taken using some criterion. If such a criterion is existent then one should determine whether it distorts the randomness of the subsamples: if you take,for instance, <strong>B</strong> from the left edge of the whole data-cloud in a scatterplot and <strong>C</strong> from the right edge then the best-fit-lines in that subsamples may have completely different slopes and variances around them. </p> <p>[update2] If such an averaging of correlations is actually meaningful in your problem (your subsamples are random and not too small) then I'd recommend to average the <a href="http://en.wikipedia.org/wiki/Fisher_transformation" rel="nofollow">z-transforms</a> of the correlation-coefficients. That means $$r_{\text{est}} = \tanh(\frac{\sum_{k=1}^{s}\tanh^{-1}(r_k)}{s})$$ where $s$ is the number of samples, because that fisher-transformation approximates by conversion a correlation-coefficient into a z-variable (normal distributed, mean=0, infinite range) where the averaging over the arithmetical mean is more meaningful. </p> <p>[update]<br> Here I show examples where the subsamples were taken randomly. I generated correlated data of a population with <em>n= 2000</em>, normal distributed with <em>mean=0, stddev=1, correlation r~ 0.35</em> . I show the variation of the occuring correlations if random samples of <em>n=20, n=50, n=100</em> are drawn. For each sample-size I did <em>500</em> experiments and documented the frequencies of occuring correlations <em>r</em> in steps of about <em>0.1</em>.</p> <pre><code>sample-n: 20 avg r: 0.37760 experiments: 500 pop-n : 2000 pop r: 0.35247 low r high r freq -------------------------------- -0.2023 -0.2023 1 -0.1807 -0.0948 8 -0.0878 0.0101 15 0.0205 0.1068 25 0.1112 0.2101 60 0.2123 0.3098 100 0.3113 0.4073 81 0.4109 0.5102 83 0.5109 0.6100 73 0.6107 0.7078 44 0.7122 0.7891 10 =================================== sample-n: 50 avg r: 0.36040 pop-n : 2000 pop r: 0.35247 low r high r freq -------------------------------- -0.1011 -0.1011 1 0.0175 0.1027 9 0.1098 0.2022 55 0.2056 0.3027 108 0.3043 0.4027 150 0.4047 0.5030 124 0.5045 0.6024 45 0.6099 0.6982 8 =================================== sample-n: 100 avg r: 0.35657 pop-n : 2000 pop r: 0.35247 low r high r freq ---------------------------------- 0.0504 0.0703 3 0.1139 0.2032 20 0.2054 0.3034 115 0.3055 0.4038 217 0.4047 0.4956 133 0.5046 0.5471 12 =================================== </code></pre> <p>One can determine confidence-intervals for the correlations; that intervals narrow with increasing size of the samples.<br> But this all is only useful if the different samples are really random and not taken by some systematic criterion.</p>