At least in $L^2$ this question has been studied in the context of Markov central limit theorems.
Let $\mathbb P_\pi$ be the induced measure on path space when starting according to $\pi$.
We have
$$\mathrm{Var}_{\mathbb P_\pi}\left( \frac 1t \int_0^t V(X_s)\,ds \right) = \frac 2t \int_0^t \left(1 - \frac rt\right)\mathrm{Cov}_{\mathbb P_\pi}[V(X_0),V(X_r)]\,dr$$
Now one needs to obtain fast enough decaying bounds for the covariances, to obtain something like $\le C/t$ as suggested in your question.
Since your state space is finite you will hopefully be able to obtain quite strong mixing results.

Geometrically fast mixing is one easy strong condition.
If you have a reversible Markov process there are also connections to the spectral gap.
Section 3.4 of these lecture notes by Eberle provides an overview.
A quick search for references on functional central limit theorems (which requires at least asymptotical results on this quantity) makes me suggest the following:

Liu, Y., Zhang, Y. Central limit theorems for ergodic continuous-time Markov chains with applications to single birth processes. *Front. Math. China* **10**, 933–947 (2015). https://doi.org/10.1007/s11464-015-0488-5