0
$\begingroup$

Problem

Consider the following data set:

YEAR;AMOUNT;MEASUREMENTS
1985;9.53013698630137;365
1986;11.086301369863;365
1987;13.0712328767123;365
1988;11.9248633879781;366
1989;10.2191780821918;365
1990;7.41933085501859;269
1991;12.1751396648045;358
1992;9.7037037037037;108
1993;13.1452261306533;199
1994;8.70697674418605;215
1995;10.5224615384615;325
1996;7.59776536312849;358
1997;10.5065753424658;365
1998;10.3983561643836;365
1999;12.971381031614;601
2000;10.3513661202186;732

The years 1990 and 1992 to 1994 have a low measurement count.

Update: Context

I am creating a system that allows the general public to create charts on climate. The purpose of the chart is to show general climate trends (through linear and non-linear regression analysis). My concern is that insufficient measurements taken throughout the year will skew the data in misleading ways.

I am using statistical analysis software for the PostgreSQL database (i.e., PL/R) to perform the calculations. I do not know if I can tell PL/R to "give less weight" to annual averages with fewer than 365 measurements. Also, if all the measurements were made in winter, then it seriously does not reflect the average maximum temperature for the year -- not even close. I also do not know if I can tell PL/R to assign weight based on when during the year the majority of measurements were made.

I have created two charts illustrating the difference between removing and keeping the sub-200 measurement counts:

Questions

  1. Should the short years be excluded from the list?
  2. If 365 measurements is the norm, what are the minimum number of measurements needed to be statistically significant (i.e., not skew the correlation)?

Thank you!

Improve this question
$\endgroup$
3
  • $\begingroup$ excluded ? Why don't you simply put less weight to these observations, instead ? $\endgroup$
    – Alekk
    Commented May 25, 2010 at 16:25
  • 4
    $\begingroup$ The answer to this question depends in an essential way on what you plan to do with the data. It would help if you added some context. $\endgroup$
    – S. Carnahan ♦
    Commented May 25, 2010 at 16:38
  • $\begingroup$ Both links to i.imgur.com are broken. I'm also unable to find any copies saved on the Wayback Machine. $\endgroup$
    – The Amplitwist
    Commented May 18, 2023 at 4:50

1 Answer 1

Reset to default
4
$\begingroup$

I cannot comment due to rep requirements but Alekk's comment reg weight- If you use hierarchical bayesian (also known as multi-level models) ideas to process the data then the observations coming from years 1990 and 1992 to 1994 will receive lower weight when estimating the parameters of interest.

In general, throwing out data is not a good idea as there is some information (perhaps it is weak) in those years.

Update/Edit

Disclaimer: I do not know anything about climate science. FWIW, my thoughts are below:

In response to your updated context, you need a way to model the climate data to account for seasonality and time trends in your data. The typical way to model time series data is to use autoregressive models. Typically, these models assume stationarity which means that the data is assumed to fluctuate around a long term average. In the context of climate science, you will have to relax this somewhat to see if the process in fact has a long term average that is increasing/decreasing. I do not think there is a canned routine that will do this for you but I could be wrong. I would encourage you to discuss the issue with a climate researcher.

Improve this answer
$\endgroup$
2
  • $\begingroup$ I switched from MySQL to PostgreSQL to use autoregressive models in PL/R. The project ends July 15th, so I don't have enough time to remodel 273 million data points. I will be speaking with a climate researcher. The data is exposed through various inputs, including starting month-day to ending month-day (e.g., March 22 - June 22). It does not make sense to divide the data categorically into seasonal constraints. The users have the ability to probe the data like that, and in many other ways. I wanted to know if it made sense statistically to toss insufficient annual data. $\endgroup$
    – Dave Jarvis
    Commented May 26, 2010 at 3:22
  • 1
    $\begingroup$ Well a properly specified AR model and Bayesian ideas will let you do one or both of the following: 1. Will accommodate years with sparse measurements appropriately and 2. Will let you impute the missing data for the years in question. The specifics of the AR model will depend on climate science and what level of aggregration you are willing to live with (e.g., do you want to model daily data, monthly data etc). $\endgroup$
    – vad
    Commented May 26, 2010 at 3:40

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .