The desired result is false for all mixing systems other than a point:
Proposition: let $T$ be an invertible totally ergodic transformation of a standard probability space $(X,\mathcal{F},\mu)$. Then there exists a measurable set $A\subset X$ such that $0<\mu(A)<1$ and
$$\mu\left(\left\{x \in X \colon T^nx \in A\text{ for all } n \in [-N,N]\right\}\right)\geq \frac{1}{2N^3+2}$$
for all $N>1$.
The proof modifies an argument of G. Hal\'asz [Remarks on the remainder in the Birkhoff ergodic theorem, Acta Mathematica Academiae Scientiarum Hungarica 28 (1976) 389-395]. The lower bound given above is somewhat arbitrary and can be modified at the expense of brevity. We use the following consequence of Rokhlin's Lemma: for every $\varepsilon>0$ and $N \geq 1$, we may find a set $B\in\mathcal{F}$ such that the sets $T^iB$ are disjoint for $|i|\leq N$, and such that their total measure is at least $1-\varepsilon$.
For each $N \geq 2$, let us choose a set $B_N$ such that the sets $T^iB_N$ are disjoint for $|i|\leq N$, and such that their union has measure at least $\frac{2N+1}{2N+2}$. In particular $\frac{1}{2N+2}\leq \mu(B_N)\leq \frac{1}{2N+1}$ since $X$ is a probability space and $T$ preserves measure. Now define
$$A:=\bigcup_{N=2}^\infty \bigcup_{|i|\leq N} T^iB_{N^3}.$$
The measure of $A$ is bounded below by $\mu(B_{2^3})>0$ and above by
$$\sum_{N=2}^\infty (2N+1)\mu(B_{N^3})\leq \sum_{N=2}^\infty \frac{2N+1}{2N^3+1}<\sum_{N=2}^\infty \frac{3N}{2N^3}=\frac{\pi^2}{4}-\frac{3}{2}<1.$$
Now note that if $x \in B_{N^3}$ then $T^ix \in T^iB_{N^3}\subset A$ when $|i|\leq N$, and therefore
$$\mu\left(\left\{x \in X \colon T^nx \in A\text{ for all } n \in [-N,N]\right\}\right)\geq \mu(B_{N^3})\geq \frac{1}{2N^3+2}$$
as claimed.