MathOverflow is a question and answer site for professional mathematicians. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have a MLP with multiple nodes in its output layer which is predicting membership of classes, one class per output node. I am currently using a "winner takes all" rule for determining which output to select as the predicted class, but intuitively I feel like classes with low membership which have a similar probability of membership for an observation should probably be chosen in favour of large classes.

Are there any accepted algorithms for choosing an output which take into consideration the class populations in the original data?

share|cite|improve this question
Simon, I don't know why you haven't got any answers yet. It may be because there are no experts round here, or because the experts don't know the answer. But one thing that might improve your chances of getting a good answer is if you clarified the question a bit for non-experts. E.g. what does MLP stand for? Can you give us some links to sources that give background on your question? Can you tell us some similar problems that have been solved? – Tom Leinster Jan 5 '10 at 13:58
I guess I was wrong there. Apparently some experts did come along; and maybe anyone capable of answering your question wouldn't benefit from the kind of clarification I was suggesting anyway. – Tom Leinster Jan 6 '10 at 16:03

Regarding your intuition, it may or may not be true. In many cases people do the opposite, i.e. downweight classes with low membership. This happens any time someone uses a prior p(c) on the class membership.

If you still think you need to boost the probability of small classes, then using different weights for different classes may help. Define the weight of class $i$ to be $W_i \sim 1/p(c_i)$ ($p(c_i)$ is the prior probability of class $i$). Then pick $i$ that minimizes something like $W_i \cdot p(c_i|MLP)$. Designing the weights has to be done empirically, I think.

For Tom: MLP is multi-layer perceptron, a type of neural network.

The problem can be reformulated as follows. Suppose you have some data (like an image of a glyph). It can belong to one of several classes (say, an English character, 'a' to 'z', so 26 classes in this case). You magically get a set of probabilities $p(c_i | data)$ (the probability that your data is in class $c_i$). In our case, this is given by the MLP. You need to decide on a single class. The most obvious rule is to pick $c_i$ for which $p(c_i | data)$ is maximal. But OP feels that if one class has very few members, it should be given advantage. Another option is to have weights and pick the class that maximizes $W_i p(c_i | data)$. The disadvantage of this is that we don't have a principled method of choosing the weights $W_i$.

One possibility is to use $W_i \sim 1/p(c_i)$. In this case, $p(c_i | data) / p(c_i) \simeq p(data | c_i)$, which seems somewhat principled (not completely arbitrary). But maybe there are better ways of doing that.

share|cite|improve this answer

Design a loss function, that is write down the cost of each type of error made by the classifier. Then choose a rule that minimizes the expected loss.

For example, you can add more penalty for incorrect prediction of outputs from rare classes.

share|cite|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.