As I indicated in my comment, Spaltenstein's monograph discusses in detail many aspects of the conjugacy classes in a disconnected reductive group. This material is not especially easy to read, partly because it's a photocopy of typescript (and in French). But he develops a lot of information about unipotent elements and their conjugacy classes in this more general framework.

Other standard sources tend to concentrate mainly on connected reductive groups, often those which are (almost) simple of Lie types $A_n, B_n, \ldots, G_2$. Besides the old Springer Lecture Notes 131 chapter by Springer and Steinberg, there is Carter's useful 1985 book *Finite Groups of Lie Type*. Carter includes a large amount of data about classes in the algebraic group and orbits in its Lie algebra, with tables and partial order diagrams for nilpotent orbits in good characteristic.
(The diagrams come from Spaltenstein's independently verified calculations, but when typeset in Carter's book those for types $E_7, E_8$ lost a few edges.)

There is also a smaller book *Nilpotent Orbits in Semisimple Lie Algebras* (1993) by Collingwood and McGovern which emphasizes the complex (and sometimes the real) case.

Lusztig-Spaltenstein duality seems to be the main focus of your question. Here the account given by Spaltenstein in his Chapter III is basic, but as he points out the determination of special unipotent classes (or nilpotent orbits in the Lie algebra) uses linear algebra in the classical types and case-by-case study in the exceptional types. The classical origin is of course the transpose duality for partitions applied to type $A_n$, where all classes or orbits are special. But here and in general the duality map is actually *order-reversing*.

The mystery about the other cases arises I think from the hidden role of Langlands duality, which interchanges types $B_n, C_n$. These groups or Lie algebras do embed in various ways in larger matrix groups or algebras, but in your formulation I don't understand how automorphisms like $\sigma_B$ would arise; semisimple automorphisms tend to have reductive (but not simple) fixed points, while nontrivial unipotent automorphisms behave quite differently.

At any rate there is not yet an intrinsic definition of what it means to be "special", since the notion arises (first with Lusztig, later with Barbasch-Vogan and others) in various kinds of representation theory. I suspect there is yet another characterization in the framework of non-restricted representations of modular Lie algebras. (Some notes on types $C_3, D_4$ are posted on my homepage.) There is already quite a bit of algebraic geometry in the approaches taken by Lusztig and Spaltenstein. It's not yet clear what method will be most enlightening and encompass the scattered occurrences of special classes/orbits.

[ADDED] My comments on duality were incomplete (but this is getting long). The further step needed to get the desired bijection of special orbits for types $B_n, C_n$ preserving both the partial order and the dimensions is explained in Spaltenstein, III.10.3-10.4. Concerning the treatment by Kac of automorphisms of finite order of semisimple Lie algebras, I'm not familiar enough with his notation to comment helpfully on how his results may fit into the framework of the question here; but keep in mind that elements of finite order (hence semisimple) of the group induce inner automorphisms whose fixed points form a reductive subgroup of maximal rank.

Classes unipotentes et sous-groupes de Borelby Nicolas Spaltenstein, where disconnected groups are emphasized and where the Lusztig-Spaltenstein duality is developed. – Jim Humphreys Jul 15 '12 at 19:11