Emil Jeřábek has already provided a detailed proof to show that the proposed theory $T$ is not bi-interpretable with $\mathsf{PA}$ (even though it is mutually interpretable with $\mathsf{PA}$).

The point of this answer is to completement Jeřábek's nice answer by indicating that the argument of Theorem 5.1 of the paper *$\omega$-models of finite set theory* (co-authored with Schmerl and Visser, available [here][1]) can be slightly fine-tuned to show the failure of bi-interpretability.  At the bottom, the argument is close in spirit to Emil's.


More specifically, the fine-tuning needed in the proof of Theorem 5.1 of the aforementioned paper to conclude the failure of bi-interpretability, is to observe that, using Remark 4.10 of the same paper, there are (many) recursive models other than $V_{\omega}$ of $\mathsf{ZF_{fin}}$ that have a definable linear ordering of order-type $\omega$, and thus they satisfy the proposed theory $T$.


  [1]: https://www.researchgate.net/publication/27715625_o-Models_of_finite_set_theory#fullTextFileContent