Here is an idea, though I have to make some assumptions.
Suppose you have two sets of points $p_1,...,p_n\in\Bbb R^d$ and $q_1,...,q_n\in\Bbb R^d$ (for example, the vertices of your polyhedra, but with a fixed order).
Assume that they are translated to be centered at the origin, i.e. $p_1+\cdots +p_n=0$, and respectively for the $q_i$, so that we can ignore translations.
In a first step you could compute the covariance matrices of both point clouds and compare them. That is
$$C_p:=\sum_{i=1}^n p_ip_i^\top,\quad C_q:=\sum_{i=1}^n q_i q_i^\top.$$
These are positive semi-definite matrices, and you can compare their lists of eigenvalues, say $\lambda_i^p$ and $\lambda_i^q$ for all $i\in\{1,...,n\}$, sorted in descending order. They tell you about how unevenly these points clouds are distributed direction-wise.
The next step is to remove this unevenness from the point clouds. If we assume that the point clouds are full-dimensional (i.e. $\mathrm{span}(p_1,...,p_n)=\Bbb R^d$), then we can define
$$p_i':=C_p^{-1/2} p_i,\qquad q_i':=C_q^{-1/2} q_i.$$
Both point sets can now no longer be distinguished by translations or directional unevenness.
The last step is to consider the correlation matrix
$$C_{pq}:=\sum_{i=1}^n p_i'q_i^{\prime \top}.$$
You could e.g. compute $\delta:=\det(C_{pq})$.
This values lies between $-1$ and $1$.
We can use it as follows:
- if $\delta=\pm1$, then the point clouds are just reorientations of each other, that is, there exists an orthogonal matrix $X\in\mathrm{O}(\Bbb R^d)$ with $\det(X)=\delta$ and $p_i=X q_i$ for all $i\in\{1,...,n\}$.
- if $\delta=0$, then these point sets are as distinct as possible.
- in general, the smaller the value of $|\delta|$, the more different these point sets are.
In the end you have to somehow use the numbes $\delta,\lambda_i^p,\lambda_i^q$ for $i\in\{1,...,n\}$ to quantify the difference between the point sets. I do not have a recipe for this. All I can tell you is, that if $\lambda_i^p=\lambda_i^q$ for all $i\in\{1,...,n\}$ and if $\delta=\pm 1$, then these point sets are the same up to some (possibly orientation-reversing) orthogonal transformation.
This of course assumes that your point sets have a predefined order (which might be given by the isomorphism between your polyhedra).
