The action is the action of the (natural) automorphism group of the relevant functor in each case. This is easiest to see for the case of
$$X^H \cong \text{Hom}_H(1, X) \cong \text{Hom}_G(G/H, X)$$
since by the Yoneda lemma the automorphism group of this functor is the automorphism group of $G/H$ as a $G$-set, which is $N_H/H$. Similarly,
$$X_H \cong X \times_H 1 \cong X \times_G G/H$$
also admits a natural action by the automorphism group of $G/H$ as a $G$-set, although I am less sure if there is a clean abstract nonsense proof that these are all the natural automorphisms. There is a universal property hiding here, which is that $G/H$ is the free $G$-set on an $H$-fixed point.
Exactly the same words can be written down for endomorphism rings instead of automorphism groups in the context of rings and modules. In the context of linear representations of groups the endomorphism ring you end up writing down is a Hecke algebra.
In more general contexts it's natural to look not only at the automorphism group or the endomorphism monoid but even the endomorphism Lawvere theory or endomorphism operad. Whereas the former give unary operations, the latter gives operations of higher arity. I give some examples here and here. Other keywords: Tannaka duality, the Barr-Beck theorem...