Take the 2-minute tour ×
MathOverflow is a question and answer site for professional mathematicians. It's 100% free, no registration required.

I'm writing some code (JavaScript) to compare benchmark results. I'm using the Welch t-test because the variance and/or sample size between benchmarks is most likely different. The critical value is pulled from a t-distribution table at 95% confidence (two-sided).

The Welch formula is pretty straight forward, but I am fuzzy on interpreting a significant result. I am not sure if the critical value should be divided by 2 or not. Help clearing that up is appreciated. Also should I be rounding the degrees of freedom, df, to lookup the critical value or would Math.ceil or Math.floor be more appropriate?

  /**
   * Determines if the benchmark's hertz is higher than another.
   * @member Benchmark
   * @param {Object} other The benchmark to compare.
   * @returns {Number} Returns `1` if higher, `-1` if lower, and `0` if indeterminate.
   */
  function compare(other) {
    // use welch t-test
    // http://frank.mtsu.edu/~dkfuller/notes302/welcht.pdf
    // http://www.public.iastate.edu/~alicia/stat328/Regression%20inference-part2.pdf
    var a = this.stats,
        b = other.stats,
        pow = Math.pow,
        bitA = a.variance / a.size,
        bitB = b.variance / b.size,
        df = pow(bitA + bitB, 2) / ((pow(bitA, 2) / a.size - 1) + (pow(bitB, 2) / b.size - 1)),
        t = (a.mean - b.mean) / Math.sqrt(bitA + bitB),
        c = getCriticalValue(Math.round(df));

    // check if t-statistic is significant
    return Math.abs(t) > c / 2 ? (t > 0 ? 1 : -1) : 0;
  }
share|improve this question
2  
I think your question is much more appropriate for stats.stackexchange.com –  Willie Wong Jan 3 '11 at 16:18
    
OK, will post there. Thanks! :D –  jdalton Jan 4 '11 at 13:17
add comment

closed as off topic by Andres Caicedo, Willie Wong, Ben Webster Jan 3 '11 at 21:06

Questions on MathOverflow are expected to relate to research level mathematics within the scope defined by the community. Consider editing the question or leaving comments for improvement if you believe the question can be reworded to fit within the scope. Read more about reopening questions here.If this question can be reworded to fit the rules in the help center, please edit the question.

Browse other questions tagged or ask your own question.