1
$\begingroup$

I am interested in doing RKHS regression with missing response variables.

Given input-output pairs $(x_i,y_i)$, I want to estimate a function $f(\cdot)$ as follows \begin{equation}f(x)\approx u(x)=\sum_{i=1}^m \alpha_i K(x,x_i),\end{equation} where $K(\cdot,\cdot)$ is a kernel function. Given that for every input $x_i$, I have an output $y_i$, the coefficients $\alpha_i$ can be found by solving \begin{equation} {\displaystyle \min _{\alpha\in R^{m}}{\frac {1}{n}}\|Y-K\alpha\|_{R^{n}}^{2}+\lambda \alpha^{T}K\alpha},\end{equation} where, with some abuse of notation, the $(i,j)$'th entry of the kernel matrix $K$ is ${\displaystyle K(x_{i},x_{j})} $. This gives \begin{equation} \alpha^*=(K+\lambda mI)^{-1}Y. \end{equation}

However, suppose now that for some inputs $x_i$, I have missing measurements $y_i$, but I would still want to estimate all weights $a_i$ in a sensible way, $i=1,..,m$. Does there exist a nice way to solve this problem?

$\endgroup$
1
$\begingroup$

This problem is called semi-supervised learning. There is ample literature on it. Essentially the points with missing value still carry information about the input distribution which you can use to improve the regression, in various ways.

$\endgroup$
0
$\begingroup$

You can use the expectation–maximization (EM) algorithm.

$\endgroup$
0
$\begingroup$

The length of my comment exceeds the limit for a comment. So, I will put it as an "answer". But please feel free to discard it. Also, please do not be bothered by the "you'' in my comments, and take them as if I were talking with you in person as your colleague :)

Here are the steps that I would try out if I were you:

(1) look into the missing mechanism for the missing $y_j$'s, and model it suitably. For a reference on how to deal with missing data, you may start with Donal B. Rubin's paper "Inference and Missing Data" at https://www.jstor.org/stable/2335739?seq=1#metadata_info_tab_contents. Donal B. Rubin is one the best minds in statistics.

Note that $y_i$'s have to have their own marginal distribution if you assume that they are identically distributed, and even if they are not, you may still be able to use the empirical CDF for the observed $y_i$'s to guess what the missing $y_j$'s may be. In the latter setting, the empirical CDF potentially has fake and wrong weights $1/n$ (with respect to Dirac masses). An understanding of the missing mechanism would allow you to adjust these weights.

(2) on the other hand, the $(x_i,y_i)$'s have to have a joint distribution if you assume that they are identically distributed. Therefore, you may be able to use the observed pairs to guess what the missing $y_i$'s are using information provided by the observed $x_i$'s. In this case, you would be estimating a conditional distribution based on the observed pairs, and then use the estimated conditional distribution to guess the missing $y_i$'s. You may use cross-validation to obtain the estimate. If $(x_i,y_i)$'s are not identically distributed, you then need to adjust the weights.

However, we often assume that $(x_i,y_i)$'s are identically distributed.

(3) in essence, guessing the missing $y_j$'s is equivalent to perturbing the objective function you wrote down, and how you would guess the $y_j$'s corresponds to the perturbation, which can be analytically written if you are willing to make simplifying assumptions. after this, you can then proceed to find all $\alpha_i$'s

(4) finally, you should follow your combined intuition as a statistician and an applied mathematician, and follow a procedure that respects the data. there is no unified or single way to deal with missing values, and we need to do it on case-by-case basis.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.