2
$\begingroup$

I have a MLP with multiple nodes in its output layer which is predicting membership of classes, one class per output node. I am currently using a "winner takes all" rule for determining which output to select as the predicted class, but intuitively I feel like classes with low membership which have a similar probability of membership for an observation should probably be chosen in favour of large classes.

Are there any accepted algorithms for choosing an output which take into consideration the class populations in the original data?

$\endgroup$
2
  • $\begingroup$ Simon, I don't know why you haven't got any answers yet. It may be because there are no experts round here, or because the experts don't know the answer. But one thing that might improve your chances of getting a good answer is if you clarified the question a bit for non-experts. E.g. what does MLP stand for? Can you give us some links to sources that give background on your question? Can you tell us some similar problems that have been solved? $\endgroup$ Jan 5, 2010 at 13:58
  • $\begingroup$ I guess I was wrong there. Apparently some experts did come along; and maybe anyone capable of answering your question wouldn't benefit from the kind of clarification I was suggesting anyway. $\endgroup$ Jan 6, 2010 at 16:03

2 Answers 2

2
$\begingroup$

Regarding your intuition, it may or may not be true. In many cases people do the opposite, i.e. downweight classes with low membership. This happens any time someone uses a prior p(c) on the class membership.

If you still think you need to boost the probability of small classes, then using different weights for different classes may help. Define the weight of class $i$ to be $W_i \sim 1/p(c_i)$ ($p(c_i)$ is the prior probability of class $i$). Then pick $i$ that minimizes something like $W_i \cdot p(c_i|MLP)$. Designing the weights has to be done empirically, I think.

For Tom: MLP is multi-layer perceptron, a type of neural network.

The problem can be reformulated as follows. Suppose you have some data (like an image of a glyph). It can belong to one of several classes (say, an English character, 'a' to 'z', so 26 classes in this case). You magically get a set of probabilities $p(c_i | data)$ (the probability that your data is in class $c_i$). In our case, this is given by the MLP. You need to decide on a single class. The most obvious rule is to pick $c_i$ for which $p(c_i | data)$ is maximal. But OP feels that if one class has very few members, it should be given advantage. Another option is to have weights and pick the class that maximizes $W_i p(c_i | data)$. The disadvantage of this is that we don't have a principled method of choosing the weights $W_i$.

One possibility is to use $W_i \sim 1/p(c_i)$. In this case, $p(c_i | data) / p(c_i) \simeq p(data | c_i)$, which seems somewhat principled (not completely arbitrary). But maybe there are better ways of doing that.

$\endgroup$
0
$\begingroup$

Design a loss function, that is write down the cost of each type of error made by the classifier. Then choose a rule that minimizes the expected loss.

For example, you can add more penalty for incorrect prediction of outputs from rare classes.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.