Here is a trial proof for the question over $Q_p$.
Write $J[f(x)^k]$ for the general Jordan form of a irreducible $f$, being $k$ identical blocks joined by 1's in general (minimal polynomial of block is $f^k$).
Let $A$ be finite order over $Q_p$, so $A\sim\oplus J[f(x)]$ where the $f$ have $f|\Phi_m$ (cyclotomic polynomials) and finiteness implies the $f$ are irreducible (not powers).
Note $\bar f$ determines $m$ up to $p$-powers, writing $m=up^v$ for $(u,p)=1$. Further note, if $\bar\Phi_u=\prod \bar g$ then $\bar\Phi_{up^v}=\prod\bar g^{\phi(p^v)}$,
and what is more, the corresponding Jordan block to $\bar g^{\phi(p^v)}$ does not split,
in other words this is the minimal polynomial. This follows since the reduction (mod $p$) of the companion matrix of $f$ is itself a companion matrix (ones above the diagonal) over a field $F_p$, and so has its minimal and characteristic polynomials equal to $\bar f=\bar g^{\phi(p^v)}$.
So, every reduction to $\bar f$ from the $A\sim\oplus J[f]$ decomposition has $\bar f(x)=\bar g(x)^{\phi(p^v)}$ for some irreducible $\bar g|\bar\Phi_u$, that lifts to $g|\Phi_u$. What is more, $\bar A\sim\oplus J[\bar g(x)^{\phi(p^v)}]$.
From this, $\bar A\sim\oplus J[\bar g(x)^{\phi(p^v)}]$ determines the general Jordan form of $A$ uniquely as something like $A\sim\oplus J[\Phi_{pu}^{g-part}(x^{p^{v-1}})]$. The general Jordan form classifies the conjugacy type over a field, as is $Q_p$.
Note that, $\Phi_3\Phi_6$ and $\Phi_6^2$ give 4x4 matrices with order 6, failing for $p=2$.