# a reliable measure of series similarity - correlation just doesnt cut it for me..

im trying to determine a method to compare one particular time series against about 10,000+ reference time series programatically.. and shortlist those reference time series which can be of interest.. the method i was using was Pearson Correlation.. for each of the reference time series, i calculate their correlation coefficients.. then i sort the entire list of reference time series in a descending order based on the correlation coefficient and visually-analyze the top N time series which have the highest correlation coefficients, and hence should be the best matches to the given time series..

trouble was that i wasnt getting reliable results.. quite often the series in the top N range didnt visually resemble anything like the given time series.. finally when i read the complete article below i understood why.. that you cant use correlation alone to determine if two time series are similar..

Anscombe's quartet

now this is problem with all matching algorithms which calculate some sort of distance between two time series.. for instance the two groups of time series below can result in the same distance yet one is obviously a better match than the other..

A  = [1, 2, 3, 4, 5, 6, 7, 8,  9]
B1 = [1, 2, 3, 4, 5, 6, 7, 8, 18]
distance = sqrt(0+0+0+0+0+0+0+0+9) = 3
B2 = [0, 3, 2, 5, 4, 7, 6, 9,  8]
distance = sqrt(1+1+1+1+1+1+1+1+1) = 3

so my question is.. is there a mathematical formula (like correlation) that can better suit me in these kind of situations? one which does not suffer from the problems mentioned here?

thanks.. =)

-
If your question does not attract useful answers here, you could consider reposting it at stats.stackexchange.com –  Yemon Choi Jan 16 '12 at 4:14
thanks.. will do that soon.. =) –  user20603 Jan 16 '12 at 5:21