I don't think this is true. Take $X = \mathbb P^1 \times \mathbb P^2$. Let $C$ be $\mathbb P^1 \times 0$, and let $C_1$ be the first infinitesimal neighborhood of $C$. The curve $C_1$ is the relative spectrum of $\mathcal O_{\mathbb P^1} \oplus O_{\mathbb P^1}^{\oplus 2}$, where $\mathcal O_{\mathbb P^1}^{\oplus 2}$ is a square-zero ideal. Any sheaf $\mathcal O_{\mathbb P^1}(d)$, with $d \ge 0$, is a quotient of $\mathcal O_{\mathbb P^1}^{\oplus 2}$. If $C(d)$ denotes the relative spectrum of $\mathcal O_{\mathbb P^1} \oplus O_{\mathbb P^1}(d)$, then $C(d)$ is contained in $C_1$ for all $d \ge 0$; hence it is embedded in $X$ with fundamental class $2[C]$. But the arithmetic genus of $C(d)$ is $-d-1$.
[Added later] For a surface, a Cohen-Macaulay curve is a divisor, and the adjunction formula shows that the arithmetic genus is determined by the cohomology class, so the answer is positive. I believe that the answer is negative for all $X$ of dimension at least three.