If there is a lot of overlap, it may be possible to get a faster algorithm, but I doubt it. If each set has one symbol that is not in the other sets, that alone will get you n! possibilities itself. If you can tell us more about the sets, we might be able to improve upon your implementation.
Gerhard "Ask Me About System Design" Paseman, 2012.08.0405