A stochastic process that is 1st and 2nd order (strictly) stationary, but not 3rd order stationary

Question

I asked this question on stats.stackexchange.com a little while back but didn't get an answer. It was suggested that I post it here at the time. There appears to be some migratory problems going on over there. Hopefully, this question is seen as appropriate.

Let $X(t)$ be a stochastic process. We say that $X(t)$ is Nth-order stationary if for every set of ''times'' $t_1, t_2, \dots, t_N$ we have that the joint cumulative density functions $$F_{X(t_1),X(t_2),\dots,X(t_N)} = F_{X(t_1 + \tau),X(t_2 + \tau),\dots,X(t_N + \tau)}$$ for all ''time shifts'' $\tau$. This is quite a strong condition, it says that the joint statistics don't change at all as time shifts.

For example, a 1st order stationary process is such that $F_{X(t_1)} = F_{X(t_2)}$ for all $t_1$ and $t_2$. That is, the $X(t)$ are all identically distributed. It is quite easy to see that a 1st order stationary process need not be 2nd order stationary. Simply assign a correlation structure to say $X(t)$, $X(t+1)$, $X(t+2)$ that does not correspond to a (symmetric) Toeplitz matrix. That is, in vector form, the covariance matrix of $[ X(t), X(t+1), X(t+3)]$ could be given as $$\left[\begin{array}{cc} \sigma^2 & a & b \newline a & \sigma^2 & c \newline b & c& \sigma^2 \end{array}\right]$$ for $a,b,c$ distinct. This is now not 2nd order stationary because $E[X(t)X(t+1)] = a$ and, time shifting by 1 we have $E[X(t+1)X(t+2)] = c \neq a$.

In a similar way (presumably) a process that is 1st and 2nd order stationary need not be 3rd order stationary and this leads to my question:

Does somebody have a nice example of a stochastic process that is both 1st and 2nd order stationary, but not 3rd order stationary?

Motivation: Some of the material I have been expected to teach recently has included stochastic processes and I feel this is a gap in my knowledge. Although no student has actually asked this question yet, I think it is natural enough for it to be asked at some point and I would like to have a neat and simple example.

Answer 1

How about this:

Consider $X_1,X_2$ i.i.d. with $\mathbb{P}(X_i = 1) = \mathbb{P}(X_i = 0) = 0.5$. Now define $X_3 = X_1 + X_2 (\text{mod }2)$. Notice that $X_3$ is independent from $X_1$ and from $X_2$ individually. However the three variables $X_1,X_2,X_3$ are not jointly independent.

Now consider a sequence $X_1,X_2,X_3,\ldots$ such that each triplet $(X_{3n+1},X_{3n+2},X_{3n+3})$ has the same distribution as $(X_1,X_2,X_3)$ and all these triplets are independent.

With this one has that all $X_n$ have the same distribution as $X_1$, so that the process is 1 stationary. Also, all pairs $X_i,X_j$ with $i\neq j$ are independent and hence have the same distribution (this gives 2 stationarity). However the triplet $(X_2,X_3,X_4)$ has a different distribution from $(X_1,X_2,X_3)$ (in fact, $X_2,X_3$ and $X_4$ are jointly independent).