4
$\begingroup$

I'm responsible for a charity donation site. We're about the change the site design, and we want to know the best way of detecting if the distribution of donations changes after the design. The problem is the data is quite clumpy, particularly around \$5, \$10, \$20, \$25 and \$50 values, with \$15 being relatively rare. There are nonetheless other real values, particularly between \$20 and \$100.

The consequence is that the stdev is three times the mean, so my first approach, a T-Test with some correction for the skew, doesn't seem feasible. If our re-design only has a moderate impact it's unlikely a T-test will be able to detect it with certainty.

Binning into \$5 bins and running a G-Test individually means I can make statements about individual categories, but I'm worried about drawing overall conclusions from such a series of measurements. I've read briefly about using Fisher's method for combining p-values but I'm not sure how to explain the result, particularly as a positive result could mean fewer \$10 donations but more \$15 donations.

I'm certain that clumpy data is a known phenomenon, but Googling hasn't helped, and my background is CS, not stats. Would anyone know the best way to handle this?

Improve this question
$\endgroup$
4
  • $\begingroup$ en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence $\endgroup$
    – Steve Huntsman
    Commented May 17, 2010 at 17:20
  • 1
    $\begingroup$ Also see en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test $\endgroup$
    – Steve Huntsman
    Commented May 17, 2010 at 17:24
  • 1
    $\begingroup$ This is not a "math" answer... but how about starting with some graphs rather than jumping to p values? For instance, after the new design has been up a week, you could make histograms of donations at different levels during each of the 10 weeks preceding the change of design, then making a histogram of the week of the new design. That would give a visual sense of significance. You also may have to factor things like different traffic patterns because of publicity related to a redesign, initial user inexperience, etc. Spending time on exploratory data analysis may make all this clearer. $\endgroup$
    – Martin M. W.
    Commented May 17, 2010 at 20:29
  • $\begingroup$ Thanks for the answers. I was aware of KL divergence having been introduced to it when doing EM, but I was ideally looking for a test that would pass with "95% confidence" or so, like T-Test. KG seems to be a promising approach, am working it through with some sample data now. $\endgroup$
    – Feenaboccles
    Commented May 19, 2010 at 15:05

1 Answer 1

Reset to default
1
$\begingroup$

The Kolmogorov–Smirnov test suggested by Steve Huntsman seems to have done the job.

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .