Let $\varphi_t : M \rightarrow M$ be a smooth flow on a smooth manifold $M$. We may assume (although I'm not sure if this is important) that the flow preserves a smooth volume form on $M$. Given a continuous function $f : M \rightarrow \mathbb{R}$, is it true that if ALL the (partial) time averages $$ A_T f(x) := {1 \over T} \int_0^T f(\varphi_t (x)) \, dt \hskip 1cm (T > 0) $$ are smooth functions, then $f$ is also smooth?

Remark. This is obviously true for the trivial action and it is non-trivially true for the case where $M = \mathbb{R}$ and the flow is the usual flow by translations. That's all I know for now.

Motivation. In the first volume of Dunford and Schwartz, where they motivate the Ergodic theorem (page 657), they state:

What is significant and measurable in the laboratory is not the quantity $f(\phi_t(x))$ but its average value $$ {1 \over T} \int_0^T f(\varphi_t (x)) \, dt $$ computed over a certain time interval $0 \leq t \leq T$.

I'm asking whether knowing the regularity of all those average values says something about the regularity of the function, or "observable" $f$. It would be interesting to consider $f$ to be just locally integrable and then the question ties in a (very) little with Wiener's differentiation theorem.