The geometry of the action of the semidirect product I'm going by the maxim

Groups, like men, are known by their actions

This naturally leads one to ask "given groups $G, H$ which act on sets $S, T$ and the semidirect product $G \rtimes H$, how does one visualize the action of $G \rtimes H$? What does it act on? Some combination of $S$ and $T$? ($S \times T$ perhaps?)
I know some elementary examples, likr $D_n \simeq \mathbb Z_n \rtimes \mathbb Z_2$. However, given an unknown situation, I am sure I cannot identify whether it is a semidirect product that is governing the symmetry.
The best responses on similar questions like intuition about semidirect product tend to refer to this as some kind of "direct product with a twist". This is shoving too much under the rug: the twist is precisely the point that's hard to visualize. Plus, not all "twists" are allowed --- only certain very constrained types of actions turn out to be semidirect product. I can justify the statement by noting that:

the space group of a crystal splits as a semidirect product iff the space group is symmorphic --- this is quite a strong rigidity condition on the set of all space groups.

This question on the natural action of the semidirect product identifies one choice of natural space for the semidirect product to act on, by introducing an unmotivated (to me) equivalence relation, which "works out" magically. What's actually going on? 
The closest answer that I have found to my liking was this one about discrete gauge theories on physics.se, where the answer mentions:

If the physical space is the space of orbits of $X$ under an action $H$. Ie, the physical space is $P \equiv X / H$. Then, if this space $P$ is acted upon by $G$. to extend this action of $G \rtimes H$ onto $X$ we need a connection.  

This seems to imply that the existence of a semidirect product relates to the ability to consider the space modulo some action, and then some action per fiber. I feel that this also somehow relates to the short exact sequence story(though I don't know exact sequences well):

Let $1 \rightarrow K \xrightarrow{f}G \xrightarrow{g}Q \rightarrow 1$ be a short exact sequence. Suppose there exists a homomorphism $s: Q \rightarrow G$ such that $g \circ s = 1_Q$. Then $G = im(f) \rtimes im(s)$. (Link to theorem) 

However, this is still to vague for my taste. Is there some way to make this more rigorous / geometric? Visual examples would be greatly appreciated. 
(NOTE: this is cross posted from math.se after getting upvotes but no answers)
 A: This is a bit of a special example in a couple ways, but maybe it might be useful for building intuition.
Suppose $1 \longrightarrow H \longrightarrow G \longrightarrow \mathbb{Z} \longrightarrow 1$ is a short exact sequence of groups. The algebraic situation you describe at the end of your question is sometimes described as saying “the sequence splits.” It’s a fact that’s not hard to prove that any sequence of the above form splits.
Algebraically, there is an element $\Phi\in\operatorname{Aut}(H)$ such that if $t$ is a generator for $\mathbb{Z}$ and $g \in H \le G$, we have $tgt^{-1} = \Phi(g)$.
Topologically, such a short exact sequence of groups corresponds to a fiber bundle over the circle, in the sense that if $E \longrightarrow S^1$ is a bundle with fiber $F$, there is such a short exact sequence with $\pi_1(F)$ playing the role of $H$ and $\pi_1(E)$ playing the role of $G$.
In good situations, (e.g. if $F$ is a $K(H,1)$), we can realize the algebraic picture topologically: if there is some map $f\colon F \to F$ such that the action of $f$ on the fundamental group yields $\Phi$, then we can build the bundle $E$ in the following way:
$$ E = (F \times [0,1])/_{(x,1) \sim (f(x),0)}.$$
In other words, $E$ is built by taking the product of $F$ with an interval (think of the interval as the “vertical” direction), and then gluing the “top” of the product to the “bottom” via $f$. We call $E$ the “mapping torus” of $f$.
Thinking geometrically may not always be clear from here. E.g. if $F$ is a manifold admitting a Riemannian metric of non-positive curvature and $f\colon F \to F$ is a diffeomorphism, the bundle $E$ may or may not also admit a Riemannian metric of non-positive curvature. The product $E = F\times S^1$ always does in this situation, of course, but for instance when $F$ is a $2$-torus, the only other examples of bundles $E$ admitting a non-positively curved metric are finitely covered by the product.
What’s going on here is that especially in low dimensions, the geometry of $E$ is really intimately tied up with the dynamics of the action of $f$ (or $\Phi$) on $F$ (or $H$).
A: Suppose $K\circlearrowright X$ is an action of $K$ on some set. We have the structure homomorphism $\varphi:K\rightarrow \text{Sym}(X).$
Let $H$ act on $K$ by automorphisms, and suppose these automorphisms can be realized as inner automorphisms within $\text{Sym}(X)$. That is, the action is given by some $\theta:H\rightarrow\text{Sym}(X)$ such that conjugations by $\theta(H)$ leave $K$ invariant. Equivalently, the $H$-action takes stabilizer subgroups of $K\circlearrowright X$ to stabilizers (it may permute them nontrivially).
Then we can construct an action of $K\rtimes H$ on $X$ by $(k,h).x = \varphi(k)\cdot\theta(h).x$, where the product $\varphi(k)\theta(h)$ is just taken in $\text{Sym}(X)$. Here, the multiplication rule for $K\rtimes H$ is
$$(k_1, h_1)\cdot (k_2,h_2) = (k_1 \cdot (h_1.k_2), h_1 h_2). $$
Any action arises in this way, since $H$ acts on $K$ by conjugation in the semidirect product $K\rtimes H$ and therefore it also acts by conjugation in the image under the structure morphism $K\rtimes H \rightarrow \text{Sym}(X)$ of an action of the semidirect product on $X$. This explains when an action of $K$ can be extended to an action of $K \rtimes H$.

It is not clear how to visualize the above. So let's pass to a nice special case.
Given another action of $H$ on a set $Y$, we can extend $H\circlearrowright Y$ to an action $K\rtimes H \circlearrowright Y$ (let the latter act via the quotient $K\rtimes H\twoheadrightarrow H$). Then we can produce an action of $K\rtimes H \circlearrowright Y \times X$. This action descends to the quotient $H\circlearrowright Y$ so that $K$ fixes each fiber $\{y\}\times X$, and $H$ acts by permuting fibers and "twisting" $X$.
If the $H$-action is faithful, the action of any element $(k,h)$ can be nicely separated into an $h$-part and a $k$-part, the $h$-part being uniquely identified by the action on $Y$. Thus given a permutation $\sigma$ of $Y\times X$ we can write that it is of the form $(k,h)$ for a known $h$, and then compute the permutation $(y,x)\mapsto \sigma\left((e,h^{-1}).(y,x)\right)$, which acts the same as $(k,h)\cdot(e,h^{-1}) = (k,e)$.
One gets some examples which appear different, but are isomorphic to these, by choosing different identifications between fibers than $\text{id}_X : \{y_1\}\times X \rightarrow \{y_2\}\times X$. These identifications may be analogous to the connection described in the question.
