I have an evaluation, where i'd like to compare two probability distributions. The easiest ways seems to compare the average. However, the standard deviation is quite high, so often the "worse" distribution is "better" in specific cases. So what would be an appropriate way to compare these results.
More concretely, i have heuristic optimization algorithms and a set of tests. Since they algorithms are heuristic and the tests contain noise, there is a lot of variance. The core question is, which algorithm is the best?
The usual formulas for confidence analysis assume that there is some "population", of which a certain percentage is polled. However, in my case there is no percentage, as the algorithms could be executed arbitrarily often.