**This is not an answer.**

Here is a possible strategy, that was too long for a comment. I briefly thought it gave a full answer, but there is a lot of stuff missing.

If $\alpha_1,\ldots,\alpha_{2g}$ are the eigenvalues on $H^1(A,\mathbb Q_\ell)$, then $\bigwedge^* H^1 = H^*$ computes all eigenvalues. Moreover, any $\frac{q}{\alpha_i}$ is another one of the $\alpha_i$ (see for example [Suh, Prop. 2.2.1]). Thus, the sum
$$\#A(\mathbb F_{q^r}) = \sum_{k=0}^{2g} (-1)^k\operatorname{tr}(\operatorname{Frob}_q^r|H^k(A,\mathbb Q_\ell))$$
is entirely determined by suitable algebraic expressions in $\alpha_1,\ldots,\alpha_g$ (w.l.o.g.).

The question then becomes the following: if $f \in \mathbb Q(x_1,\ldots,x_g)$ is a symmetric rational function, does knowing $f(\alpha_1^r,\ldots,\alpha_g^r)$ for $r = 1,\ldots,g$ determine $\alpha_1,\ldots,\alpha_g$ up to permutation? For $f = x_1+\ldots+x_g$, the answer is yes by classical theory, and it seems plausible that therefore the answer is yes for general $f$. But our specific function may be bad.

[Suh] <cite authors="Suh, Junecue">_Suh, Junecue_, [**Symmetry and parity in Frobenius action on cohomology**](http://dx.doi.org/10.1112/S0010437X11007056), Compos. Math. 148, No. 1, 295-303 (2012). [ZBL1258.14023](https://zbmath.org/?q=an:1258.14023).</cite>