This is a very open-ended question, which may or may not have a perfect answer, and for which I have a few ideas but nothing like a clear picture. However, I guess it won't hurt to ask to see if people think about such things at all and if they do, what their ideas are. I don't know whether to make it CW or not: on one hand, it is pure mathematics, so we are within the usual set of standards to judge what's right and what's wrong, on the other hand, it is certainly not "a question of the type MO was designed for". So, I'm hesitant to check the community wiki option myself but have absolutely nothing against someone else doing so.
I assume that the ancient Greeks had an idea of a complete normed space ($\mathbb R$ and $\mathbb R^2$ would be enough for our purposes for quite a while), a set, a linear transformation, and the center of mass. On the top of it, I assume they had as much common sense (probably more), as we have nowadays.
The task is the usual one for Archimedes: given a reasonable non-empty set $E$ in a complete linear space $V$, assign a point $C(E)$ to it that you can confidently call "the center of mass". For the purposes of this thread, let's consider bounded at most countable subsets in $V=\mathbb R$ first. If we can figure out what to do with this case to everyone's satisfaction, we can move to the next stage. It may be not a really illuminating model, but it has a few quite funny features already.
The axioms of the center of mass are just the common sense ones:
1) The center of mass is never outside the closed convex hull of the set.
2) If $A$, $B$ are disjoint, then $C(A\cup B)\in[C(A),C(B)]$.
3) If $T$ is an affine transformation, then $C(TE)=TC(E)$.
4) (this is a bit tough, so feel free to drop or to modify it if it helps) If $A,B$ are such that the sets $a+B$, $a\in A$ are disjoint, then $C(A+B)=C(A)+C(B)$
I don't know if we really need anything else (in particular, I'm not sure if the addition of the "obvious" definition of the center of mass of a finite set is needed, helpful, or hurtful), but feel free to play with this list in any reasonably way you want.
The questions are the usual ones:
C) Way to find $C(E)$ given $E$.
Any ideas, constructions, counterexamples, references, etc. (not necessarily restricted to the model I described) are welcome :).
Edit: Thanks to everyone who responded! Let me clarify one thing. Yes, if we can create a meaningful notion of mass (say, if there exists a translation invariant Borel measure that is finite and positive on $E$), we can define the center of mass in the usual way and the only question will be if the definition is unambiguous. However, if I give you a symmetric set, you'll not hesitate to say that the center of symmetry is also the center of mass. Also, if I give you an infinite set and add one point to it, you'll probably (but not necessarily) agree that the center of mass won't feel this addition, the reason being that the set was infinitely times more massive than the point we added not as mush because we can measure the actual mass in some way but merely because infinitely many disjoint shifts of the point fit inside the set. In other words, quite often we can determine the relative size without being able to assign any meaning to the absolute size. This is one of the loopholes in the integration theory I'd like to exploit and see how far one can get with it. In a sense, that is a straight extension of the original Eudoxus line of thinking, which is why "ancient Greeks" entered the title of the question.