Stack Exchange network consists of 183 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.
Equality holds (meaning $\langle f, A_{N,M} f \rangle \to \mu(A)^2$) for every $A$ if and only if the system is ergodic, meaning there are no $T$-invariant sets $A$ having $0<\mu(A)<1.$ To see that equality can fail when $T$ is not ergodic, let $(X,\mu)$ be a standard probability space and $T$ the identity transformation. If $\mu(A)=1/2$, then the limit in question is $1/2$, not $1/4$.
When $\mu$ is a probability measure, setting $h\equiv 1$, Cauchy-Schwarz implies $\int g \cdot \bar{h}\, d\mu \leq \Bigl(\int |g|^2\, d\mu\, \int |h|^2\, d\mu\Bigr)^{1/2} $. Squaring both sides, simplifying using $h\equiv 1$ and the fact that $\mu$ is a probability measure produces the desired inequality.
I had some success reading the article Values of quadratic forms at primitive integral points by Dani and Margulis, which proves a slightly stronger version of Oppenheim, and in particular implies that the values of the specific quadratic form in the questions are dense. As to @Ycor's question, these notes by Dani have a section on 'Quantitative Oppenheim', and a Google search for that phrase leads to a plethora of results. Unfortunately I don't know what the state of the art is.
A remark on the circle rotation example (and uniquely ergodic topological systems in general): for everywhere convergence to hold, the boundary of $A$ must have measure $0$.
However, for group rotations $(G,m_G, R_\alpha)$ and continuous functions $f$, it is the case that $n \mapsto f(R_\alpha^n x)$ is uniformly almost periodic for every $x\in G$.
In the measure preserving setting, there are discrete spectrum transformations $T$ and bounded functions $f$ such that for all $x$, the map $n\mapsto f(T^n x)$ is not almost periodic. For example, take an irrational circle rotation, and let $f$ be the characteristic function of a compact totally disconnected set having positive positive measure. Then for almost all $x$, the set $\{n : f(T^n x) > 1/2\}$ is nonempty and is not syndetic, which is enough to show that $n\mapsto f(T^n x)$ is not uniformly almost periodic.
I think this description of $C_u(\mathbb R)$ says the extreme points of the unit ball of $C_u(\mathbb R)^*$ are finitely additive $\{0,1\}$-valued measures supported on cosets of $\mathbb Z.$ This makes a bit of sense, since an atomless $\{0,1\}$-valued finitely additive measure supported on $\{n+\frac{1}{n}:n\in \mathbb N\}$ induces an element of $C_u(\mathbb R)^*$ which comes from such a measure supported on $\mathbb N,$ thanks to uniform continuity.