What is the symmetry group of this compound of two polytopes?

Question

The geometric shape in question is a compound of two polytopes: an 11-hypercube with edge length $2$ and an 11-simplex with edge length $\sqrt6$ whose vertices are a subset of the hypercube’s. What is the structure of this shape’s symmetry group?

Edit: proof that it’s possible: https://www.math.uchicago.edu/~may/VIGRE/VIGRE2011/REUPapers/Markov.pdf

And explicit coordinates:

(–1,–1,–1,–1,–1,–1,–1,–1,–1,–1,–1) (1,–1,1,–1,–1,–1,1,1,1,–1,1) (1,1,–1,1,–1,–1,–1,1,1,1,–1) (–1,1,1,–1,1,–1,–1,–1,1,1,1) (1,–1,1,1,–1,1,–1,–1,–1,1,1) (1,1,–1,1,1,–1,1,–1,–1,–1,1) (1,1,1,–1,1,1,–1,1,–1,–1,–1) (–1,1,1,1,–1,1,1,–1,1,–1,–1) (–1,–1,1,1,1,–1,1,1,–1,1,–1) (–1,–1,–1,1,1,1,–1,1,1,–1,1) (1,–1,–1,–1,1,1,1,–1,1,1,–1) (–1,1,–1,–1,–1,1,1,1,–1,1,1)

Answer 1

The automorphism group of this configuration $C'$ is the Mathieu group $M_{11}$.

Firstly, we construct a larger configuration $C$ consisting of a 12-dimensional orthoplex inscribed in a 12-dimensional hypercube.

In particular, the orthoplex in $C$ consists of the vectors $\{ \pm v : v \textrm{ is a column of } H \}$, where $H$ is a 12-by-12 Hadamard matrix whose first row is $(1, 1, \dots, 1)$. The hypercube in $C$ simply consists of all $2^{12}$ vectors whose entries are $\pm 1$.

The automorphism group of $C$ contains the order-2 centre $\{\pm I\}$, modulo which it is isomorphic to the Mathieu group $M_{12}$. This is described here in terms of the [unique up to isomorphism] order-12 Hadamard matrix:

http://www.maths.qmul.ac.uk/~lsoicher/designtheory.org/library/encyc/topics/had.pdf

The two different 12-element permutation representations of $M_{12}$ correspond to permuting:

1. The 12 'coordinate axes' of the hypercube;
2. The 12 pairs of opposite vertices of the inscribed orthoplex;

The subgroup $K$ that fixes a particular one of the 12 coordinate axes also has the order-2 centre $\{ \pm I \}$, modulo which it is isomorphic to $M_{11}$. This follows from $M_{11}$ being describable as the stabiliser of a point in the permutation representation of $M_{12}$.

Indeed, $K$ factors as a direct product $C_2 \times M_{11}$, where the first factor indicates whether the first coordinate axis is flipped or not, and the second factor indicates the permutation of the remaining 11 coordinate axes.

Consequently, $M_{11}$ is precisely the subgroup of the symmetry group of $C$ which fixes the unit vector $v_1 = (1, 0, \dots, 0)$.

Finally, note that the OP's configuration $C'$ can be described as the restriction of $C$ to the 11-dimensional hyperplane $\{ x . v_1 = 1 \}$. (The 11-simplex is obtained as a facet of the 12-orthoplex, and the 11-hypercube is a facet of the 12-hypercube.) Another construction of $C'$ is to take the vertices of the simplex to be the columns of the rectangular matrix $H'$ obtained by deleting the first row from $H$; the vertices of the hypercube are again the $\pm 1$-vectors.

It follows, therefore, that the symmetry group of $C'$ is $M_{11}$. This has an 11-element permutation representation (permuting the coordinate axes of the hypercube) and a 12-element permutation representation (permuting the vertices of the simplex).