1
$\begingroup$

I'll start with a motivating example and only then proceed to the question.

Consider a list of total packages of milk that were purchased on 9 consecutive days on a given store,

$z_1,\ldots,z_9 = 1,0,0,2,0,1,0,1,3$

Assume I have an algorithm that predicts $\hat{z}_{10}=2$. I'm interested in assessing the confidence associated to this prediction, however, for stock management purposes, I want to measure the confidence that the predicted value is an upper bound on the true value $z\leq\hat{z}$, instead of $z=\hat{z}$. This is because what I'm ultimately after is to be confident in not running out of stock.

I guess an estimate of this confidence, based only on given examples, is $1-p_{\hat{z}_{10}}$ where $p_{\hat{z}_{10}}$ is the probability of finding a value more extreme than the prediction $\hat{z}_{10}$ on the empirical distribution of all 10 observations:

X
X X
X X X
X X X X
-------
0 1 2 3

which would be $1-p_{\hat{z}_{10}}=0.9$.

Assuming the above reasoning is correct (enough :), now to the question:

I recently read "A Tutorial on Conformal Prediction" by Shafer and Vovk, and am curious on how to frame this problem on the Conformal Prediction framework. It appears to me the paper exclusively focus the case of estimating confidence for $z=\hat{z}$, so the question how to adapt it for the asymmetric case.

If we were interested in the $z=\hat{z}$ case, a natural nonconformity measure would be $A(B,\hat{z})=|\bar{z}_B-\hat{z}|$, where $\bar{z}_B$ is the mean of $B$ (section 4.1 in the paper). That would define the following prediction regions (from the algorithm of section 4.2):

$\Gamma^{0\leq\varepsilon<0.5}=\{0,1,2,3\}$

$\Gamma^{0.5\leq\varepsilon<1}=\{1,2\}$

$\Gamma^{1}=\{\}$

given that,

$\alpha_0=A(\{1,2,3\},0)=\alpha_3=A(\{0,1,2\},3)=2$

$\alpha_1=A(\{0,2,3\},1)=\alpha_2=A(\{0,1,3\},2)=2/3$

$p_0=p_3=0.5$

$p_1=p_2=1$

It feels wrong to transport this reasoning for the asymmetric case, as it considers both directions (e.g. 0) as "more extreme". In particularly would mean that we have a 0.5 confidence in the $z\leq\bar{z}_{10}$ prediction, much lower than 0.9 which was given above from the empirical distribution.

It appears to me that for solving this problem we have to choose a nonconformity measure $A(B,\bar{z})$ that is monotonic with regard to $\bar{z}$, for example, $A(B,\bar{z})=\bar{z}$. That would make "more extreme" asymmetric, resulting in the following prediction regions for the above example:

$\Gamma^{0\leq\varepsilon<0.1}=\{0,1,2,3\}$

$\Gamma^{0.1\leq\varepsilon<0.3}=\{0,1,2\}$

$\Gamma^{0.3\leq\varepsilon<0.6}=\{0,1\}$

$\Gamma^{0.6\leq\varepsilon<1}=\{0\}$

$\Gamma^{1}=\{\}$

given that,

$\alpha_i=A(B,i)=i$

$p_0=1$

$p_1=0.6$

$p_2=0.3$

$p_3=0.1$

This means that we can be at least 0.7 confident in the the $z\leq\bar{z}_{10}$ prediction. Notably it still does not match the 0.9 confidence obtained from the empirical distribution. I wonder if I am on the right track...

Thanks!

Marco

$\endgroup$
2
$\begingroup$

Marco, thanks for contacting me by email.

It is possible to modify your nonconformity measure $A(B,z)=|\bar z_B - z|$ to solve your problem: it needs to be made "one-sided", so that only large values are regarded as strange. Namely, you can use $A(B,z)=z-\bar z_B$. You suggest using $A(B,z)=z$ instead; this is a smart move since, while giving the same prediction intervals, it simplifies the calculations greatly. Let me go through the calculations for $A(B,z)=z$ (similar but messier calculations for $A(B,z)=z-\bar z_B$ are a useful exercise). You use sets in your calculations, but it is important to use multisets. In "A Tutorial on Conformal Prediction" we use a special notation for multisets but here I will simply write $\{\ldots\}$; just remember that there can be several entries of the same element. If the test object is $z$, the nonconformity scores are: $$ \alpha_{10}=z, \alpha_1=\alpha_6=\alpha_8=1, \alpha_2=\alpha_3=\alpha_5=\alpha_7=0, \alpha_4=2, \alpha_9=3. $$ Let me assume that $z$ is a nonnegative real number (so that we have amounts of milk rather than packages); this will also cover your discrete case. If we increase $z$ from 0 to $\infty$, the p-value will change when $z$ crosses 1, 2, and 3. Namely, the p-values are: $0.1$ when $z\in(3,\infty)$, $0.2$ when $z\in(2,3]$, $0.3$ when $z\in(1,2]$, $0.6$ when $z\in(0,1]$, $1$ when $z=0$. This gives these prediction intervals: $\Gamma^{0\le\epsilon<0.1}=[0,\infty)$ (vacuous), $\Gamma^{0.1\le\epsilon<0.2}=[0,3]$, $\Gamma^{0.2\le\epsilon<0.3}=[0,2]$, $\Gamma^{0.3\le\epsilon<0.6}=[0,1]$, $\Gamma^{0.6\le\epsilon<1}=\{0\}$, $\Gamma^{1}=\emptyset$. This is similar to what you get but not identical; e.g., $\Gamma^0$ should be $\{0,1,2,3,4,\ldots\}$ in your discrete case, not just $\{0,1,2,3\}$ (the sales can exceed what you have seen so far.)

Does it make sense to you?

$\endgroup$
  • $\begingroup$ It does perfect sense, I see now where my mistake was. Thank you very much for the clarification! $\endgroup$ – mvc Aug 7 '17 at 11:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.