This is not an answer.
Here is a possible strategy, that was too long for a comment. I briefly thought it gave a full answer, but there is a lot of stuff missing.
If $\alpha_1,\ldots,\alpha_{2g}$ are the eigenvalues on $H^1(A,\mathbb Q_\ell)$, then $\bigwedge^* H^1 = H^*$ computes all eigenvalues. Moreover, any $\frac{q}{\alpha_i}$ is another one of the $\alpha_i$ (see for example [Suh, Prop. 2.2.1]). Thus, the sum $$\#A(\mathbb F_{q^r}) = \sum_{k=0}^{2g} (-1)^k\operatorname{tr}(\operatorname{Frob}_q^r|H^k(A,\mathbb Q_\ell))$$ is entirely determined by suitable algebraic expressions in $\alpha_1,\ldots,\alpha_g$ (w.l.o.g.).
The question then becomes the following: if $f \in \mathbb Q(x_1,\ldots,x_g)$ is a symmetric rational function, does knowing $f(\alpha_1^r,\ldots,\alpha_g^r)$ for $r = 1,\ldots,g$ determine $\alpha_1,\ldots,\alpha_g$ up to permutation? For $f = x_1+\ldots+x_g$, the answer is yes by classical theory, and it seems plausible that therefore the answer is yes for general $f$. But out specific function may be bad.
[Suh] Suh, Junecue, Symmetry and parity in Frobenius action on cohomology, Compos. Math. 148, No. 1, 295-303 (2012). ZBL1258.14023.).