I'll focus on potential topological characterizations of the action. As I mentioned in the comments above, every element of $G$ will either perserve some Borel probability measure on $\partial{T}$, or will have north-south dynamics.

If the group $G$ acts discretely, cocompactly on $\mathcal{T}$, then there's Bowditch's characterization that this is equivalent to $G$ acting properly dicontinuously and cocompactly on the triple points in $\partial{T}$. In fact, in this case $G$ will be a virtually free hyperbolic group, and a (finite) graph of finite groups.

I believe that $Aut(\mathcal{T})$ is closed in $Aut(\partial{\mathcal{T}})$, with the induced topology. In this case, one may assume that $G\leq Aut(\mathcal{T})$ is closed, otherwise taking its closure $\overline{G}$ in $Aut(\partial{\mathcal{T}})$ will correspond to taking its closure in $Aut(\mathcal{T})$, and if $\overline{G}\leq Aut(\mathcal{T})$, then $G\leq Aut(\mathcal{T})$. So assume now that $G$ is closed in $Aut(\partial{\mathcal{T}})$.

If $G$ is a compact subgroup of $Aut(\partial{\mathcal{T}})$, then I think it should be a profinite group, which acts on a tree (elliptically with a global fixed point). However, I'm not sure exactly how to tell if the action of $G$ on $\partial{\mathcal{T}}$ corresponds to this group action. Let's assume from now on that $G$ is not compact.

If $G$ does not act cocompactly on $\mathcal{T}$, then consider the limit set $\Lambda(G)\subset \partial{\mathcal{T}}$, which is the set of accumulation points of $Gx\subset \partial{\mathcal{T}}$ for any $x\in\partial{\mathcal{T}}$. One may realize this as the closure of the limit points of hyperbolic elements of $G$. Then $G$ should act cocompactly on the convex hull of $\Lambda(G)=\mathcal{R}$ inside of $\mathcal{T}$. So one may replace $\partial{\mathcal{T}}$ with $\partial{\mathcal{R}}=\Lambda(G)$.

So now assume that $G$ is a closed noncompact subgroup of $Aut(\partial{\mathcal{T}})$, and $\Lambda(G)=\partial{\mathcal{T}}$. Now, I believe that there should be a generalization of Bowditch's theorem. $G$ should be hyperbolic as a topological group, generated by a compact subset. I think this should be equivalent to $G$ acting properly cocompactly on the triple point space of $\partial{\mathcal{T}}$, but I don't know if this is proven (or even correct). Compare to the action of $Isom(\mathbb{H}^n)$ on $(\partial\mathbb{H}^n)^3-\Delta$, which is proper and cocompact.

There is another possible topological characterization in terms of actions on spaces of walls, but this is really just an encoding of the tree on $\partial{T}$ in terms of how each edge partitions $\partial{T}$ into pairs of clopen sets. If $G$ is closed, but noncompact and nondiscrete, then one might be able to encode the tree by the maximal compact subgroups of $G$ and their intersections, essentially Bass-Serre theory. But I haven't thought this through.

I'll focus on potential topological characterizations of the action. As I mentioned in the comments above, every element of $G$ will either lie in a maximal compact subgroup of $Aut(\partial{T})$, or will have north-south dynamics (a hyperbolic element).

If the group $G$ acts discretely, cocompactly on $\mathcal{T}$, then there's Bowditch's characterization that this is equivalent to $G$ acting properly dicontinuously and cocompactly on the triple points in $\partial{T}$. In fact, in this case $G$ will be a virtually free hyperbolic group, and a (finite) graph of finite groups.

I believe that $Aut(\mathcal{T})$ is closed in $Aut(\partial{\mathcal{T}})$, with the induced topology. In this case, one may assume that $G\leq Aut(\mathcal{T})$ is closed, otherwise taking its closure $\overline{G}$ in $Aut(\partial{\mathcal{T}})$ will correspond to taking its closure in $Aut(\mathcal{T})$, and if $\overline{G}\leq Aut(\mathcal{T})$, then $G\leq Aut(\mathcal{T})$. So assume now that $G$ is closed in $Aut(\partial{\mathcal{T}})$.

If $G$ is a compact subgroup of $Aut(\partial{\mathcal{T}})$, then I think it should be a profinite group, which acts on a tree (elliptically with a global fixed point). However, I'm not sure exactly how to tell if the action of $G$ on $\partial{\mathcal{T}}$ corresponds to this group action. Let's assume from now on that $G$ is not compact.

If $G$ does not act cocompactly on $\mathcal{T}$, then consider the limit set $\Lambda(G)\subset \partial{\mathcal{T}}$, which is the set of accumulation points of $Gx\subset \partial{\mathcal{T}}$ for any $x\in\partial{\mathcal{T}}$. One may realize this as the closure of the limit points of hyperbolic elements of $G$. Then $G$ should act cocompactly on the convex hull of $\Lambda(G)=\mathcal{R}$ inside of $\mathcal{T}$. So one may replace $\mathcal{T}$ with $\mathcal{R}$, and $\partial{\mathcal{T}}$ with $\partial{\mathcal{R}}=\Lambda(G)$.

So now assume that $G$ is a closed noncompact subgroup of $Aut(\partial{\mathcal{T}})$, and $\Lambda(G)=\partial{\mathcal{T}}$. Now, I believe that there should be a generalization of Bowditch's theorem. $G$ should be hyperbolic as a topological group, generated by a compact subset. I think this should be equivalent to $G$ acting properly cocompactly on the triple point space of $\partial{\mathcal{T}}$, but I don't know if this is proven (or even correct). Compare to the action of $Isom(\mathbb{H}^n)$ on $(\partial\mathbb{H}^n)^3-\Delta$, which is proper and cocompact.

There is another possible topological characterization in terms of actions on spaces of walls, but this is really just an encoding of the tree on $\partial{T}$ in terms of how each edge partitions $\partial{T}$ into pairs of clopen sets. If $G$ is closed, but noncompact and nondiscrete, then one might be able to encode the tree by the maximal compact subgroups of $G$ and their intersections, essentially Bass-Serre theory. But I haven't thought this through.

I'll focus on topological characterizations of the action. As I mentioned in the comments above, every element of $G$ will either perserve some Borel probability measure on $\partial{T}$, or will have north-south dynamics.

If the group $G$ acts discretely, cocompactly on $\mathcal{T}$, then there's Bowditch's characterization that this is equivalent to $G$ acting properly dicontinuously and cocompactly on the triple points in $\partial{T}$. In fact, in this case $G$ will be a virtually free hyperbolic group, and a (finite) graph of finite groups.

I believe that $Aut(\mathcal{T})$ is closed in $Aut(\partial{\mathcal{T}})$, with the induced topology. In this case, one may assume that $G\leq Aut(\mathcal{T})$ is closed, otherwise taking its closure $\overline{G}$ in $Aut(\partial{\mathcal{T}})$ will correspond to taking its closure in $Aut(\mathcal{T})$, and if $\overline{G}\leq Aut(\mathcal{T})$, then $G\leq Aut(\mathcal{T})$. So assume now that $G$ is closed in $Aut(\partial{\mathcal{T}})$.

If $G$ is a compact subgroup of $Aut(\partial{\mathcal{T}})$, then I think it should be a profinite group, which acts on a tree (elliptically with a global fixed point). However, I'm not sure exactly how to tell if the action of $G$ on $\partial{\mathcal{T}}$ corresponds to this group action. Let's assume from now on that $G$ is not compact.

If $G$ does not act cocompactly on $\mathcal{T}$, then consider the limit set $\Lambda(G)\subset \partial{T}$, which is the set of accumulation points of $Gx\subset \partial{T}$ for any $x\in\partial{T}$. One may realize this as the closure of the limit points of hyperbolic elements of $G$. Then $G$ should act cocompactly on the convex hull of $\Lambda(G)=\mathcal{R}$ inside of $\mathcal{T}$. So one may replace $\partial{\mathcal{T}}$ with $\partial{\mathcal{R}}$, which is determined by the limit set of $G$.

So now assume that $G$ is a closed noncompact subgroup of $Aut(\partial{\mathcal{T}})$, and $\Lambda(G)=\partial{\mathcal{T}}$. Now, I believe that there should be a generalization of Bowditch's theorem. $G$ should be hyperbolic as a topological group, generated by a compact subset. I think this should be equivalent to $G$ acting properly cocompactly on the triple point space of $\partial{\mathcal{T}}$, but I don't know if this is proven (or even correct).

There is another possible topological characterization in terms of actions on spaces of walls, but this is really just an encoding of the tree on $\partial{T}$ in terms of how each edge partitions $\partial{T}$ into pairs of clopen sets. If $G$ is closed, but noncompact and nondiscrete, then one might be able to encode the tree by the maximal compact subgroups of $G$ and their intersections, essentially Bass-Serre theory. But I haven't thought this through.

I'll focus on potential topological characterizations of the action. As I mentioned in the comments above, every element of $G$ will either perserve some Borel probability measure on $\partial{T}$, or will have north-south dynamics.

If the group $G$ acts discretely, cocompactly on $\mathcal{T}$, then there's Bowditch's characterization that this is equivalent to $G$ acting properly dicontinuously and cocompactly on the triple points in $\partial{T}$. In fact, in this case $G$ will be a virtually free hyperbolic group, and a (finite) graph of finite groups.

I believe that $Aut(\mathcal{T})$ is closed in $Aut(\partial{\mathcal{T}})$, with the induced topology. In this case, one may assume that $G\leq Aut(\mathcal{T})$ is closed, otherwise taking its closure $\overline{G}$ in $Aut(\partial{\mathcal{T}})$ will correspond to taking its closure in $Aut(\mathcal{T})$, and if $\overline{G}\leq Aut(\mathcal{T})$, then $G\leq Aut(\mathcal{T})$. So assume now that $G$ is closed in $Aut(\partial{\mathcal{T}})$.

If $G$ is a compact subgroup of $Aut(\partial{\mathcal{T}})$, then I think it should be a profinite group, which acts on a tree (elliptically with a global fixed point). However, I'm not sure exactly how to tell if the action of $G$ on $\partial{\mathcal{T}}$ corresponds to this group action. Let's assume from now on that $G$ is not compact.

If $G$ does not act cocompactly on $\mathcal{T}$, then consider the limit set $\Lambda(G)\subset \partial{\mathcal{T}}$, which is the set of accumulation points of $Gx\subset \partial{\mathcal{T}}$ for any $x\in\partial{\mathcal{T}}$. One may realize this as the closure of the limit points of hyperbolic elements of $G$. Then $G$ should act cocompactly on the convex hull of $\Lambda(G)=\mathcal{R}$ inside of $\mathcal{T}$. So one may replace $\partial{\mathcal{T}}$ with $\partial{\mathcal{R}}=\Lambda(G)$.

So now assume that $G$ is a closed noncompact subgroup of $Aut(\partial{\mathcal{T}})$, and $\Lambda(G)=\partial{\mathcal{T}}$. Now, I believe that there should be a generalization of Bowditch's theorem. $G$ should be hyperbolic as a topological group, generated by a compact subset. I think this should be equivalent to $G$ acting properly cocompactly on the triple point space of $\partial{\mathcal{T}}$, but I don't know if this is proven (or even correct). Compare to the action of $Isom(\mathbb{H}^n)$ on $(\partial\mathbb{H}^n)^3-\Delta$, which is proper and cocompact.

There is another possible topological characterization in terms of actions on spaces of walls, but this is really just an encoding of the tree on $\partial{T}$ in terms of how each edge partitions $\partial{T}$ into pairs of clopen sets. If $G$ is closed, but noncompact and nondiscrete, then one might be able to encode the tree by the maximal compact subgroups of $G$ and their intersections, essentially Bass-Serre theory. But I haven't thought this through.