I'll start with a motivating example and only then proceed to the question.

Consider a list of total packages of milk that were purchased on 9 consecutive days on a given store,

$z_1,\ldots,z_9 = 1,0,0,2,0,1,0,1,3$

Assume I have an algorithm that predicts $\hat{z}_{10}=2$. I'm interested in assessing the confidence associated to this prediction, however, for stock management purposes, I want to measure the confidence that the predicted value is an upper bound on the true value $z\leq\hat{z}$, instead of $z=\hat{z}$. This is because what I'm ultimately after is to be confident in not running out of stock.

I guess an estimate of this confidence, based only on given examples, is $1-p_{\hat{z}_{10}}$ where $p_{\hat{z}_{10}}$ is the probability of finding a value more extreme than the prediction $\hat{z}_{10}$ on the empirical distribution of all 10 observations:

```
X
X X
X X X
X X X X
-------
0 1 2 3
```

which would be $1-p_{\hat{z}_{10}}=0.9$.

Assuming the above reasoning is correct (enough :), now to the question:

I recently read "A Tutorial on Conformal Prediction" by Shafer and Vovk, and am curious on how to frame this problem on the Conformal Prediction framework. It appears to me the paper exclusively focus the case of estimating confidence for $z=\hat{z}$, so the question how to adapt it for the asymmetric case.

If we were interested in the $z=\hat{z}$ case, a natural nonconformity measure would be $A(B,\hat{z})=|\bar{z}_B-\hat{z}|$, where $\bar{z}_B$ is the mean of $B$ (section 4.1 in the paper). That would define the following prediction regions (from the algorithm of section 4.2):

$\Gamma^{0\leq\varepsilon<0.5}=\{0,1,2,3\}$

$\Gamma^{0.5\leq\varepsilon<1}=\{1,2\}$

$\Gamma^{1}=\{\}$

given that,

$\alpha_0=A(\{1,2,3\},0)=\alpha_3=A(\{0,1,2\},3)=2$

$\alpha_1=A(\{0,2,3\},1)=\alpha_2=A(\{0,1,3\},2)=2/3$

$p_0=p_3=0.5$

$p_1=p_2=1$

It feels wrong to transport this reasoning for the asymmetric case, as it considers both directions (e.g. 0) as "more extreme". In particularly would mean that we have a 0.5 confidence in the $z\leq\bar{z}_{10}$ prediction, much lower than 0.9 which was given above from the empirical distribution.

It appears to me that for solving this problem we have to choose a nonconformity measure $A(B,\bar{z})$ that is monotonic with regard to $\bar{z}$, for example, $A(B,\bar{z})=\bar{z}$. That would make "more extreme" asymmetric, resulting in the following prediction regions for the above example:

$\Gamma^{0\leq\varepsilon<0.1}=\{0,1,2,3\}$

$\Gamma^{0.1\leq\varepsilon<0.3}=\{0,1,2\}$

$\Gamma^{0.3\leq\varepsilon<0.6}=\{0,1\}$

$\Gamma^{0.6\leq\varepsilon<1}=\{0\}$

$\Gamma^{1}=\{\}$

given that,

$\alpha_i=A(B,i)=i$

$p_0=1$

$p_1=0.6$

$p_2=0.3$

$p_3=0.1$

This means that we can be at least 0.7 confident in the the $z\leq\bar{z}_{10}$ prediction. Notably it still does not match the 0.9 confidence obtained from the empirical distribution. I wonder if I am on the right track...

Thanks!

Marco