Gaussian process kernel parameter tuning I am reading on gaussian processes and there are multiple resources that say how the parameters of the prior (kernel, mean) can be fitted based on data,specifically by choosing those that maximize the marginal likelihood. However, if we use an expression of our data to fit the parameters of the prior, doesnt this defeat the point of a prior? Wouldnt it be the same to fit the parameters that best explain the data and use them directly, instead of bayesian inference?
Thank you for your time
 A: I do not know a mathmatical answer, but can give some intuition from the side of (Bayesian) machine learning.
If possible, one would like to avoid fitting parameters at all, and instead use strict Bayesian inference to condition models on data points. Luckily, this is possible for Gaussian processes. Hence, Gaussian process models do not seem to overfit in practice, if one has a good Gaussian process prior.
Sadly, this Bayesian approach is not possible for many other models. The, parameters might be treated by other clever means (MCMC, variational inferences, ...) or fitted by maximizing a likelihood. All of these methods tend to approximate strict Baysian inference. Maximizing the likelihood tends to induce strong overfitting if many parameters are fitted using few data points.
For Gaussian processes, one can somewhat successfully fit parameters of the prior by maximizing the likelihood. This might (and sometimes does) lead to overfitting. But since the parameters of the prior are one more level removed from the data, the overfitting effect is usually much smaller in cmparison to fitting model parameters.
