Here is a proof for the question over $Q_p$.
Write $J[f(x)^k]$ for the general Jordan form of a irreducible $f$, being $k$ identical blocks joined by 1's in general (minimal polynomial of block is $f^k$).
Let $A$ be finite order over $Q_p$, so $A\sim\oplus J[f(x)]$ where the $f$ have $f|\Phi_m$ (cyclotomic polynomials) and finiteness implies the $f$ are irreducible (not powers).
Note $\bar f$ determines $m$ up to $p$-powers, writing $m=up^v$.
Furthernote, $\bar\Phi_u\rightarrow\oplus J[\bar g(x)]$ gives $\bar\Phi_{up^v}\rightarrow\oplus J[\bar g(x)^{\phi(p^v)}]$ where $\phi(p^v)\neq 1$ as $p\neq 2$. That is, the minimal polynomial of a block of $\bar\Phi_{up^v}$ corresponding to powers of $\bar g$ is as large as possible, namely $\bar g^{\phi(p^v)}$.
So, every reduction to $\bar f$ from the $A\sim\oplus J[f]$ decomposition, has $\bar f(x)=\bar g(x)^{\phi(p^v)}$ for some irreducible $\bar g|\bar\Phi_u$, that lifts to $g|\Phi_u$.
From this, $\bar A\sim\oplus J[\bar g(x)^{\phi(p^v)}]$ determines the general Jordan form of $A$ uniquely as $A\sim\oplus J[g(x^{\phi(p^v)})]$. This classifies it over a field, like $Q_p$.
Note that, $\Phi_3\Phi_6$ and $\Phi_6^2$ give 4x4 matrices with order 6, failing for $p=2$.