Suppose we are given a set S of points on which two different groups G and G' (given by sets of generating permutations) act. Is there an efficient algorithm for finding generators the largest pair of subgroups H and H' of G and G' whose action on S coincide?

This paper studies the (easier) problem of checking if the intersection of two subgroups of $S_n$ is trivial. In particular, it is shown that the graph isomorphism problem polynomially reduces to that problem. The converse reduction is not known. Thus the problem of checking that the intersection is trivial is at least as hard as the graph isomorphism problem. It is difficult but not known to be NPhard. Update Also look at this ICM talk by Babai. 


It is known that the following three computational problems for subgroups $G$ of $S_n$ are polynomially equivalent:
As Mark says, these are all at least as difficult as graph isomorphism. The proofs are clever but basically elementary and interesting, so I recommend them! One reference is: E.M. Luks, ``Permutation groups and polynomialtime computation'', in L. Finkelstein and W.M. Kantor (eds.), Groups and Computation, Dimacs Series in Discrete Mathematics and Theoretical Computer Science vol. 11, American Math. Soc., 139176, 1993. I have just noticed that Luks has a recently published book with the same title, which I have not seen yet. Added later: It should also be mentioned that the implementations of the above algorithms in GAP and Magma involve backtrack searches, and so are potentially exponential, but in practice they run fast for most examples of moderate degree. 

