0
$\begingroup$

I have 3 neural networks processing 3 different vectors of values. Each NN processes a sample of it's vector and gives binary result (y/n) that is correct with given probability. All 3 NNs give answer to the same question. The task is to combine this results into single one (y/n) and figure out probability that it is correct.

What methods can you suggest for this task?

I found random effects model, but i can't apply it to my task because it operates with means and variances while my data is different: it's a binary answer with probability of correctness.

Thanks in advance.

Improve this question
$\endgroup$
0

1 Answer 1

Reset to default
2
$\begingroup$

Assume that the networks err independently and the corresponding probabilities are $(TP)_j, (TN)_j, (FP)_j, (FN)_j$ (false negative = the answer is yes and the network outputs no, etc.). Also assume that the price you pay for an error of each type is the same for both answers.

Suppose that you get YNY, say. This means either Yes and $(TP)_1(FN)_2(TP)_3$, or No and $(FP)_1(TN)_2(FP)_3$. The combined output should correspond to the higher product, taking into account the prevalence $P$ of the Yes answer, so if you have $P(TP)_1(FN)_2(TP)_3>(1-P)(TP)_1(FN)_2(TP)_3$, you settle on Yes, otherwise on No.

You cannot beat the (properly understood) maximal likelihood under any circumstances and this case is simple enough to match it if you know the prevalence and all 3 two by two detection tables. After that it is easy to figure out the detection table for the combined method by a straightforward casework.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Huge thanks. It helped a lot. Can i ask you to give a bit more explanation about detection tables? I'm sorry, i didn't get the point. Thanks. $\endgroup$
    – yaromir
    Jul 5, 2014 at 23:22
  • $\begingroup$ In general, there are 4 probabilities for any testing device with y/n output: true positive (probability to detect a present thing (threat, disease, dependence, whatever), true negative (not detecting an absent thing), false positive (detecting something that is not there) and false negative (failing to detect something that is there). Ideally, the device should have tp=tn=1, fp=fn=0, but that never happens. There are two obvious relations: tp+fn=tn+fp=1, but otherwise 4 numbers are independent. I call the whole set of these values "detection table", though it is not a standard name $\endgroup$
    – fedja
    Jul 6, 2014 at 0:36
  • $\begingroup$ @yaromir The classical names for the two independent parameters are sensitivity and specificity, but to be honest, I never remember what exactly those are and have to google them every time. The point however is that the error of saying "yes" when the answer is "no" may be very different from the error of saying "no" when the answer is "yes" in terms of both the probability and the cost of erring. So I just decided to cover more than you formally asked for. $\endgroup$
    – fedja
    Jul 6, 2014 at 0:41
  • $\begingroup$ And one more minor question. Shouldn't there be $P(TP)_1(FN)_2(TP)_3>(1-P)(FP)_1(TN)_2(FP)_3$ instead of $P(TP)_1(FN)_2(TP)_3>(1-P)(TP)_1(FN)_2(TP)_3$ ? If i understood correctly, left part represents probability that answer that was given by majority of NNs is correct while right part is a probability that despite that majority of the NNs gave this answer this major answer is wrong (false answer of the whole system). $\endgroup$
    – yaromir
    Jul 15, 2014 at 16:39

Not the answer you're looking for? Browse other questions tagged or ask your own question.