(1) Which graph classes are extremely tough to test for graph non-isomorphic pairs from isomorphic pairs?
(2) Is there a repository of adjacencies from such classes?
There is a move towards creating a "standard" benchmark set for graph isomorphism, but it didn't happen yet. Meanwhile, the largest collection that includes hard graphs as well as easy graphs is here. To make examples of difficult nonisomorphic pairs, take two graphs with the same parameters and randomly label them. For difficult isomorphic pairs take two random labellings of the same graph.
The graphs here that will cause trouble to most programs are in the classes ag, cfi, had, latin-sw, pg, pp, f-lex.
In ScrewBox: a Randomized Certifying Graph-Non-Isomorphism Algorithm by Kutz and Schweitzer it is stated that "It is generally accepted that the incidence graphs of finite projective planes confront graph isomorphism algorithms with great challenges."
Also the PhD Thesis of Schweitzer discusses difficult graphs (including incidence graph of projective planes) in section 2.8.
I do not know any repository of adjacencies of these graph, but Moorhouse has data online for known projective planes of order 27 online. Also the paper of Kutz and Schweitzer as well as the thesis of Schweitzer both cite data that Royle has on projective planes on order 16 but that link seems to broken for me right now.
Paper: A large database of graphs and its use for benchmarking graph isomorphism algorithms
The graph instances and graph generator software are here: