To answer the rephrased question: is there a simple lower-bound for the rate of convergence? Yes: assuming certain conditions hold, which I describe below.
For any $t \ge 0$, let $\Pi_t(x, \cdot)$ denote the transition probability of the process $X_t$ defined as: $$
\Pi_t(x,A) = \mathbb{P}(X_t \in A ~\mid~ X_0= x )
$$
for any measurable set $A \subset D$. It is sufficient to establish a Doeblin condition, which is not impractical to obtain since the domain of the diffusion is bounded. This condition requires that there exist $t \ge 0$, a constant $\alpha \in (0,1)$ and a probability measure $\nu$ on $D$ such that
$$
\inf_{x \in D} \Pi_t(x, A) \ge \alpha \nu(A) \;.
$$
for all measurable sets $A \subset D$. If this Doeblin condition holds, then one can apply a coupling inequality to show that the rate of convergence of $\Pi_t(x,\cdot)$ to equilibrium is determined by the parameter $\alpha$ appearing in this bound.
The usual way to guarantee that this condition holds is to show that $X_t$ is irreducible and its associated semigroup is strongly Feller, see e.g. Section 4 of Hairer 2001. To prove that a diffusion on $\mathbb{R}^d$ is irreducible and its associated semigroup is strongly Feller, it is sufficient to assume that the drift field $\mu$ of the diffusion is locally Lipschitz continuous with at most polynomial growth at infinity, see Chapter 2 of Second Order PDE's in Finite and Infinite Dimension. S. Cerrai. Springer, 2001.
On a bounded domain, and depending on the assumed boundary conditions, other assumptions may be needed to prove irreducibility and the strong Feller property, including a regularity assumption on the boundary, e.g., that the boundary is twice differentiable; for precise conditions see Chapter 4 of Linear and Quasilinear Equations of Parabolic Type. Ladyzensakaja et al. American Mathematical Society, 1967 or other textbooks that discuss existence and properties of classical solutions to linear PDEs of parabolic type on bounded domains.
Note that, in contrast to diffusions on $\mathbb{R}^d$, for bounded diffusions one does not need to assume that the drift is dissipative. In fact, it suffices to assume enough regularity on $\mu$ and $\partial D$ such that the above Doeblin condition holds.