Rather then directly considering the invariant differential operators, you might want to consider its functional calculus only. This can be realized/studied via convolution products and representation theory. What I describe is in the realm of the first answer by Pedro Lauridsen Ribeiro. I let you decide whether this classifies as "algebraic", but it is certainly of operator-algebraic/ representation-theoretic flavour.

Here is an example. Assume $H$ is compact. We identify $L^2(G/H)$ with the induced representation $\pi = Ind_{H}^{G} 1$ or the $H$-invariant vectors in $L^2(G)$ and then uses the convolution operators for $\phi \in C_c^\infty(G//H)$: $$T_\phi f(g) = \int\limits_{G} \phi(x) f(xg) d x.$$ E.g. for $G =SL_2(\mathbb{R})$ and $H=SO(2)$, the algebra $C_c^\infty(G//H)$ is commutative by the Gelfand trick (this is not so essential) and the trace $T_\phi$ is an integral of the Harish-Chandra/Selberg transform of $\phi$ over the spectrum of the hyperbolic Laplacian or, alternatively, an integral $$ \int\limits tr\; \pi(\phi) d_{Pl} \pi$$ over the irreducible unitary (tempered) reps $\pi$ of $G$ with $H$-invariant vectors (only principal series representations here). The measure $d_{Pl}$ is the Plancherel measure.

If $H$ is not compact, you can still do something similar working with $C_c^\infty(G)$ and obtain a similar analysis. E.g. take the two important situations when $H$ is a lattice or a parabolic subgroup in a reductive Lie group. The advantage: this generalizes to locally compact groups. E.g. on reductive groups over non-archimedean fields, there are no differential operators in any obvious way, but this gives you the Hecke operators. These ideas are crucial also in the context of the Selberg trace formula.

Rather then only considering invariant PSD operators, you might want to consider all $G$ invariant operators, i.e., $G$-intertwiner. I describe their functional calculus below. Their functional calculus can be realized/studied via convolution products and representation theory. What I describe is in the realm of the first answer by Pedro Lauridsen Ribeiro. I let you decide whether this classifies as "algebraic", but it is certainly of operator-algebraic/ representation-theoretic flavour. I claim everything else which is $G$-invariant operator will have a equivalent functional calculus.

Here is an example. Assume $H$ is compact. We identify $L^2(G/H)$ with the induced representation $\pi = Ind_{H}^{G} 1$ or the $H$-invariant vectors in $L^2(G)$ and then uses the convolution operators for $\phi \in C_c^\infty(G//H)$: $$T_\phi f(g) = \int\limits_{G} \phi(x) f(xg) d x.$$ E.g. for $G =SL_2(\mathbb{R})$ and $H=SO(2)$, the algebra $C_c^\infty(G//H)$ is commutative by the Gelfand trick (this is not so essential) and the trace $T_\phi$ is an integral of the Harish-Chandra/Selberg transform of $\phi$ over the spectrum of the hyperbolic Laplacian or, alternatively, an integral $$ \int\limits tr\; \pi(\phi) d_{Pl} \pi$$ over the irreducible unitary (tempered) reps $\pi$ of $G$ with $H$-invariant vectors (only principal series representations here). The measure $d_{Pl}$ is the Plancherel measure.

If $H$ is not compact, you can still do something similar working with $C_c^\infty(G)$ and obtain a similar analysis. E.g. take the two important situations when $H$ is a lattice or a parabolic subgroup in a reductive Lie group. The advantage: this generalizes to locally compact groups. E.g. on reductive groups over non-archimedean fields, there are no differential operators in any obvious way, but this gives you the Hecke operators. These ideas are crucial also in the context of the Selberg trace formula.