It turns out that $\mathcal{F}_\infty$ is never trivial (at least when the distribution of $X_s$ has a second moment). This can be seen by calculating the covariance of the average $Y_s$ at generation $n$ with $Y_r$, and comparing it to the variance of this average.