Let $X$ and $Y$ be sets. It is undecidable in ZFC whether $2^{|X|} = 2^{|Y|}$ implies $|X| = |Y|$ (in Cohen's original model for ZFC + $\neg$CH, one has $2^{\aleph_0} = 2^{\aleph_1}$). What if we restrict our attention to the finite parts of $X$ and $Y$?

Question. Do $X$ and $Y$ have the same cardinality if the families of finite subsets of both sets do?

As noted by Nik Weaver in a comment, the answer is yes when we assume in the background to be working with the axioms of ZFC. But, what about ZF?

I feel this must be either basic or very well known (to those who know it very well).