Perhaps instead of finite sets one should work with effective divisors, i.e., formal linear combibinations of the form $\newcommand{\bZ}{\mathbb{Z}}$ $\newcommand{\bR}{\mathbb{R}}$ $\newcommand{\bp}{\boldsymbol{p}}$
$$ D= \sum_{\bp\in\bR^N} m_D(\bp) \bp, $$
where $m_D:\bR^N\to\bZ_{\geq 0}$ is a function with finite support. Denote by $\newcommand{\Div}{\mathrm{Div}}$ $\Div_{\geq 0}(\bR^N)$ the space of effective divisors in $\bR^N$. $\Div_{\ge 0}$ has an obvious semigroup structure. Think of the center of mass as a map $\newcommand{\eC}{\mathscr{C}}$
$$ \eC: \Div_{\geq 0}(\bR^N)\to \Div_{\geq 0}(\bR^N), $$
with the following properties.
1. For any divisor $D$, the support of the center of mass $\eC(D)$ consists of a single point.
2. For any divisors $D_1,D_2\in\Div_{\geq 0}(\bR^N)$ we have
$$ \eC(D_1+D_1) =\eC\Bigl(\; \eC(D_1)+\eC(D_2)\; \Bigr).$$
3. For any point $\bp\in \bR^N$
$$\eC(\delta_{\bp})=\delta_{\bp}, $$
where $\delta_{\bp}$ denotes the Dirac divisor of mass $1$ supported at $\bp$.
4. $\eC$ is continuous with respect to the obvious topology on $\Div_{\geq 0}(\bR^N)$, where supports of divisors converge in the Hausdorff metric and in the limit the total mass is conserved.
Claim. I believe that these properties uniquely determine $\eC$.
Weaker Claim. The map $\eC$ is uniquely determined if besides 1,...4 we assume
5. if the Support of $D$ is contained in an affine subspace $V\subset \bR^N$, then the support of $eC(D)$ is contained in the same affine subspace.