Let's say we have a Markov process $X_t$, and we come up with a forecast model that takes some information from outside world and says: "value $X_{t+1}$ has probability density distribution $P_t(x)$". The forecasted distribution changes on every step, because it depends on some parameters from outside world that change on every step. We look at each consequtive $X_t$, and try to compare it to previously forecasted $P_t(x)$ every time. How do we say that the underlying model is valid?
$\begingroup$
$\endgroup$
6
-
$\begingroup$ Well, if your predicted distros are all gaussian you could just rescale and recenter everybody and plot the histogram to check whether it s accurate. The next step is doing model comparison $\endgroup$– Guillaume DehaeneCommented Aug 10, 2015 at 15:37
-
$\begingroup$ Distributions are a bit more complex, like power tail. Isn't there a probabilistic methodology in general ("good" enough though) case? $\endgroup$– mt_christoCommented Aug 10, 2015 at 17:39
-
$\begingroup$ Can you maybe scale them back so that they are similar or something like that ? Otherwise, you ll have to go to model comparaison which will require a simple model to compare your proposal to. Look up bayesian model comparaison $\endgroup$– Guillaume DehaeneCommented Aug 10, 2015 at 19:59
-
$\begingroup$ Guillaume, it's impossible to scale to similarity in my case, unfortunately - I want to track errors caused by wrong tail/mid scale shapes, and they are different even after being re-scaled (different tail power, for example). The family is still pretty simple (like Pareto - symmetric and good, with finite variances for my case, etc.), but rescaling is not applicable. Narrow case, but not as narrow as Gaussian. Do I understand it correctly, that there's no conservative probabilistic methodology around judgement of such varying distribution forecasts? $\endgroup$– mt_christoCommented Aug 11, 2015 at 10:33
-
$\begingroup$ Not that I know of, but that's not saying much ^^ Even if you can't rescale your predictions to a single shape, one thing you can do is check if some important statistics have been predicted well. Note $\mu_t, \sigma_t$ the prediction for the mean and std of $X_t$. Then the values $X_t - \mu_t$ should have empirical mean 0, and $(X_t - \mu_t)^2 / \sigma^2_t $ should have empirical mean 1. You can check those predictions (and similar predictions for the higher moments). If applicable, you can also check if the CLT applies and gives you the correct result. $\endgroup$– Guillaume DehaeneCommented Aug 11, 2015 at 14:27
|
Show 1 more comment