There are centrally symmetric self-dual polytopes in every dimension. This follows from Proposition 3.9 in
Reisner, S., Certain Banach spaces associated with graphs and CL-spaces with 1- unconditional bases, J. Lond. Math. Soc., II. Ser. 43, No. 1, 137-148 (1991). ZBL0757.46030.
Moreover, in dimension $\geqslant 3$ the matrix $X$ can be chosen to be a permutation matrix.
Here is an example in dimension $3^d$ for every $d$. Start with Sztencel-Zaramba polytope $P$. This is the unit ball for the norm on $\mathbf{R}^3$
$$ \|(x,y,z)\| = \max \left( |y|+|z|, |x|+\frac 12 |z| \right)$$
whose dual norm satisfies
$$ \|(x,y,z)\|_* = \|(z,y,x)\|. $$
We may now define inductively a sequence $\|\cdot\|_d$, which is norm on $\mathbf{R}^{3^d}$ (identified with $\mathbf{R}^{3^{d-1}}\times\mathbf{R}^{3^{d-1}}\times\mathbf{R}^{3^{d-1}}$). Chose $\|\cdot\|_1$ to be above norm, and use the recursive formula
$$ \|(x,y,z)\|_{d+1} = \|( \|x\|_d ,\|y\|_d , \|z\|_d )\|_1 .$$
One checks by induction that there is a permutation matrix which maps the unit ball onto its polar.
To visualize the polytope $P$ you may use the Sage code
p1 = Polyhedron(vertices=[[0,1,1],[0,1,-1],[0,-1,1],[0,-1,-1],[1,0,1/2],[1,0,-1/2],[-1,0,1/2],[-1,0,-1/2]])
p1.projection().plot()