# Is there a way of canonically labelling permutation groups?

When working with large numbers of graphs, a canonical labelling routine is essential as, after the initial cost of canonically labelling each graph, it permits isomorphism checks to be replaced with identity checks.

Currently I am working with large numbers of transitive permutation groups; the degree is small (40ish) but the size of group can be large, and my programs are gumming up due to having to perform large numbers of - not exactly isomorphism - but conjugacy checks. (I need to check each pair of groups to see if they are conjugate in the symmetric group.)

This process would be considerably easier if there was a known way of finding a canonical conjugate of the incoming permutation groups...

-
the question of finding a canonical form of the centraliser ring of a permutation group is already non-trivial to me. I do not know anything better than trying all the orderings of orbitals, canonising each edge-coloured graph you get this way, and pick the maximal. (if the orbitals are "different" in some way, e.g. have different subdegrees, you might use it to restrict to, say, ordering the orbitals wrt subdegrees) The centraliser ring in a canonical form would give you a natural fingerprint for conjugacy. –  Dima Pasechnik Mar 4 '12 at 4:46
I'd also look into what actually happens when you check the conjugacy with your software (GAP? Magma?). Maybe it's doing something dumb... –  Dima Pasechnik Mar 4 '12 at 4:50
Dima, I'm using Magma, but I do not know of any way to determine (directly) whether it is doing anything dumb. I will try and do some experiments and see whether it is doing certain obvious things. –  Gordon Royle Mar 6 '12 at 8:02
Do you actually need canonization, or would a complete invariant suffice for your purposes? Although the two are computationally equivalent for graph isomorphism (there is a polynomial-time canonical labelling algorithm for graphs iff there is a poly-time complete invariant), this seems not to be true in general. So a complete invariant might be easier to find. –  Joshua Grochow Jul 2 '13 at 20:28

Babai, Codenotti, and Qiao (ICALP 2012 from Babai's homepage, also in Codenotti's thesis) show how to test permutational isomorphism (=conjugacy - but this might be another good phrase to search with) of permutation groups in time that is polynomial in the order of the groups and and simply exponential ($c^n$ for fixed $c$) in the size of the permutation domain. From a theoretical perspective, I believe this is the best known.
In the transitive case, the runtime is still simply exponential in the degree but only linear in the order, and in this amount of time they can in fact list all of the permutational isomorphisms. Listing the permutational isomorphisms allows you to pick a canonical labelling by, e.g., lexicographic order. (And $|G|2^{40}$ is way better than the naive bound of $|G|40!$: $2^{40} \approx 10^{12}$, while $40! \approx 10^{48}$.)
Now theoretical bounds on algorithms may not tell you anything about efficiency in practice. (Depending on how large your transitive groups are, even the linear factor of $|G|$ might be too much for you.) It is at least reasonable to think that their algorithm is not only theoretically more efficient but also more efficient in practice than the naive algorithm(s). And of course one can always hope that it will work for your instances anyways, or perhaps with a few additional heuristics.
This seems to be one of the biggest holes in the practical isomorphism arsenal. Various non-trivial possibilities come to mind, but their efficiency relies a lot on a more basic question. Define (and efficiently compute) a total order on labelled finite permutation groups. Thus, for any two permutation groups $G_1,G_2$ given by permutation generators, determine whether $G_1\lt G_2$, $G_1=G_2$ or $G_1\gt G_2$, where $G_1=G_2$ iff they are exactly the same (not just isomorphic) and the order obeys the transitivity law. This is possible in principle: compute the entire groups and compare them lexicographically. But naturally we don't want to compute the entire groups.