Perhaps instead of  finite sets one should work with *effective divisors*, i.e., formal linear combibinations of the form $\newcommand{\bZ}{\mathbb{Z}}$ $\newcommand{\bR}{\mathbb{R}}$ $\newcommand{\bp}{\boldsymbol{p}}$

$$ D= \sum_{\bp\in\bR^N} m_D(\bp) \bp,  $$

where $m_D:\bR^N\to\bZ_{\geq 0}$ is a function  with finite support. Denote by $\newcommand{\Div}{\mathrm{Div}}$  $\Div_{\geq 0}(\bR^N)$ the space of effective divisors in $\bR^N$.  $\Div_{\ge 0}$ has an obvious semigroup structure. Think of the center of mass as a map $\newcommand{\eC}{\mathscr{C}}$

$$ \eC: \Div_{\geq 0}(\bR^N)\to \Div_{\geq 0}(\bR^N),  $$

such  that for any divisor $D$ the support of the center of math $\eC(D)$ consists of a single point.  One important property  of this map is

$$ \eC(D_1+D_1) =\eC\Bigl(\; \eC(D_1)+\eC(D_2)\; \Bigr).$$

Another property is

$$\eC(\delta_{\bp})=\delta_{\bp}, $$

where $\delta_{\bp}$ denotes the Dirac divisor of mass $1$ supported at  $\bp$.

These two properties uniquely determine  $\eC$.  To extend it to more general  mass distributions one needs  sime continuity assumptions on $\eC$.