show/hide this revision's text 2 added 180 characters in body; added 3 characters in body

I've been asking myself this question all the time. Let's say you are given a large set of time series data. Your task is to find out patterns that are meaning meaningful or that you can use for future trend prediction.

The issue now is, how do you know for sure that the patterns you extract are valid, in the sense that they don't suffer from data snooping bias or a case of "torture-the-data-until-it-confesses"?

I can always test my hypothesis as new data comes in, but even if it can predict all the trends in the past, that doesn't mean that it will continue to do so in the future. No?
show/hide this revision's text 1

Data Mining-- How do You Know Whether The Pattern You Extract is Valid?

I've been asking myself this question all the time. Let's say you are given a large set of time series data. Your task is to find out patterns that are meaning or that you can use for future trend prediction.

The issue now is, how do you know for sure that the patterns you extract are valid, in the sense that they don't suffer from data snooping bias or a case of "torture-the-data-until-it-confesses"?