a reliable measure of series similarity - correlation just doesnt cut it for me..

Question

im trying to determine a method to compare one particular time series against about 10,000+ reference time series programatically.. and shortlist those reference time series which can be of interest.. the method i was using was Pearson Correlation.. for each of the reference time series, i calculate their correlation coefficients.. then i sort the entire list of reference time series in a descending order based on the correlation coefficient and visually-analyze the top N time series which have the highest correlation coefficients, and hence should be the best matches to the given time series..

trouble was that i wasnt getting reliable results.. quite often the series in the top N range didnt visually resemble anything like the given time series.. finally when i read the complete article below i understood why.. that you cant use correlation alone to determine if two time series are similar..

Anscombe's quartet

now this is problem with all matching algorithms which calculate some sort of distance between two time series.. for instance the two groups of time series below can result in the same distance yet one is obviously a better match than the other..

A  = [1, 2, 3, 4, 5, 6, 7, 8,  9]
B1 = [1, 2, 3, 4, 5, 6, 7, 8, 18]
     distance = sqrt(0+0+0+0+0+0+0+0+9) = 3
B2 = [0, 3, 2, 5, 4, 7, 6, 9,  8]
     distance = sqrt(1+1+1+1+1+1+1+1+1) = 3

so my question is.. is there a mathematical formula (like correlation) that can better suit me in these kind of situations? one which does not suffer from the problems mentioned here?

thanks.. =)

Answer 1

If you wish to stay model-free, then interquartile range and rank correlation are much more robust measures. Obviously with 10,000 candidate series interquartile range is too coarse a measure by itself so you would need an objective function comprised of more than just that metric. I like using a weighted combination of rank correlation and interquartile range.

Because your data are time series data, you may achieve better results by differencing your data points first (or, for price data, even turning them into a return series).

With so many candidate series, you are likely to end up with false positives. In the usual kinds of ensemble analysis this is addressed by parameter shrinking. That's more difficult with robust quantile-based measures of course, but you can get a sense for the reliability of your matches by running against 10,000 (or more) randomly generated series.

I try to avoid Pearson correlation these days: its popularity has more to do with its computational tractability than its relevance to most real-world situations. Modern computers make it possible to work with robust measures fairly easily.