4
$\begingroup$

I am reading on gaussian processes and there are multiple resources that say how the parameters of the prior (kernel, mean) can be fitted based on data,specifically by choosing those that maximize the marginal likelihood. However, if we use an expression of our data to fit the parameters of the prior, doesnt this defeat the point of a prior? Wouldnt it be the same to fit the parameters that best explain the data and use them directly, instead of bayesian inference?

Thank you for your time

Improve this question
$\endgroup$
5
  • 1
    $\begingroup$ To improve your chances of getting an answer, make your post self-contained. Doing so, use LaTeX/MathJax. $\endgroup$
    – Iosif Pinelis
    Commented Apr 8, 2021 at 15:38
  • $\begingroup$ Have you tried asking this question in MSE? (I would suggest maybe also adding the context of that article into your post, since it may be too much to ask from users to read that post before giving an answer). $\endgroup$
    – Pedro
    Commented Apr 8, 2021 at 15:39
  • $\begingroup$ thank you, i ll rephrase my question. However, i dont see how MSE is relevant. I am mainly talking about bayesian prior parameter fitting $\endgroup$
    – john
    Commented Apr 8, 2021 at 17:09
  • $\begingroup$ The procedure you are describing is referred to as "empirical Bayes". Consider for example the lengthscale parameter of a kernel. An more Bayesian treatment might be to say: I don't know the lengthscale I expect the problem to have, but I expect it to be broadly in this range, and define a 'hyperprior'. Then do inference over it. But why stop there? We could have a hyper-hyper prior. Pragmatically, we have to stop at some point and provide some sort of a value, and one way to obtain it would be to look at the data. $\endgroup$
    – Oxonon
    Commented May 12, 2021 at 17:33
  • $\begingroup$ One thing to note is that the kernel hyperparameters that we typically tune have relatively little effect on the properties of a Gaussian process estimator. Once you pick a specific kernel function, that defines the function space you are considering --- changing the tuneable hyperparameters is generally a reweighting of elements in that space (in the sense that the prior over functions that are induces with difference hyperparameters tend to be absolutely continuous wrt each other). $\endgroup$
    – Oxonon
    Commented May 12, 2021 at 17:36

1 Answer 1

Reset to default
1
$\begingroup$

I do not know a mathmatical answer, but can give some intuition from the side of (Bayesian) machine learning.

If possible, one would like to avoid fitting parameters at all, and instead use strict Bayesian inference to condition models on data points. Luckily, this is possible for Gaussian processes. Hence, Gaussian process models do not seem to overfit in practice, if one has a good Gaussian process prior.

Sadly, this Bayesian approach is not possible for many other models. The, parameters might be treated by other clever means (MCMC, variational inferences, ...) or fitted by maximizing a likelihood. All of these methods tend to approximate strict Baysian inference. Maximizing the likelihood tends to induce strong overfitting if many parameters are fitted using few data points.

For Gaussian processes, one can somewhat successfully fit parameters of the prior by maximizing the likelihood. This might (and sometimes does) lead to overfitting. But since the parameters of the prior are one more level removed from the data, the overfitting effect is usually much smaller in cmparison to fitting model parameters.

Improve this answer
$\endgroup$
1
  • $\begingroup$ I am aware of the overfitting effect, thus i was curious about why contaminate the prior with data at all, since the point of the prior is to encode our assumptions a priori. Otherwise, a prior with data is called a posterior. But it seems to be a very common practise nonetheless. My own intuition is the same as yours, i.e. avoid fitting parameters at all $\endgroup$
    – john
    Commented Apr 16, 2021 at 11:50

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .