The action is the action of the (natural) automorphism group of the relevant functor in each case. This is easiest to see for the case of 

$$X^H \cong \text{Hom}_H(1, X) \cong \text{Hom}_G(G/H, X)$$

since by the Yoneda lemma the automorphism group of this functor is the automorphism group of $G/H$ as a $G$-set, which is $N_H/H$. Similarly,

$$X_H \cong X \times_H 1 \cong X \times_G G/H$$

also admits a natural action by the automorphism group of $G/H$ as a $G$-set, although I am less sure if there is a clean abstract nonsense proof that these are all the natural automorphisms. There is a universal property hiding here, which is that $G/H$ is the free $G$-set on an $H$-fixed point.

Exactly the same words can be written down for endomorphism rings instead of automorphism groups in the context of rings and modules. In the context of linear representations of groups the endomorphism ring you end up writing down is a Hecke algebra. 

In more general contexts it's natural to look not only at the automorphism group or the endomorphism monoid but an even larger structure, namely the endomorphism Lawvere theory. Whereas the former give unary operations, the latter gives operations of higher arity. I give some examples <a href="http://qchu.wordpress.com/2013/06/09/operations-and-lawvere-theories/">here</a> and <a href="http://qchu.wordpress.com/2013/06/23/operations-pro-objects-and-grothendiecks-galois-theory/">here</a>. Other keywords: <a href="http://ncatlab.org/nlab/show/Tannaka+duality">Tannaka duality</a>, the <a href="http://ncatlab.org/nlab/show/monadicity+theorem">Barr-Beck theorem</a>...