Question

Let us define $d:X^n\rightarrow\mathbb R$. How can we define metric properties such as symmetry, triangle inequality equivalent property etc for such a function?

Answer 1

One should let the properties follow from an interpretation. We didn't write down the axioms for a metric space because they're intrinsically interesting: we wrote them down because they describe the notion of distance.

One thing that certainly has been tried is to let $d(x,y,z)$ describe the area of a triangle between the three points $x,y,z$. Then the appropriate axioms are that $d$ be completely symmetric (swapping any two of $x$, $y$ and $z$ gives the same answer) and that $$d(x,y,z) \leq d(a,y,z) + d(x,a,z) + d(x,y,a).$$ In general, one could let $d(x_0,\ldots,x_n)$ be thought of as the $n$-volume of an $n$-simplex. Then the appropriate axioms are complete symmetry again, and a very similar triangle inequality (where the right-hand-side is the sum of what you get by substituting $a$ for each $x_i$).

Answer 2

Here are some comments about why one might be interested in generalizing metrics to multiple inputs. These comments, however, seem somewhat orthogonal to what the OP is looking for.

1. Intuitive motivation: metrics are intimately related to means. So if we are willing to consider means of more than two objects at a time, we can surely consider "metric"-like multiargument functions.

2. A more practical motivation is as follows. A metric allows us to measure a "distance" between two points. In many applications, the absolute distance is less important than relative distance. That is, questions such as $d(a,b) < d(a,c)$ are more important. Another way of speaking this is: the singleton $\lbrace a\rbrace$ is "closer" to the singleton $\lbrace b\rbrace$ than to $\lbrace c \rbrace$. This view, immediately suggests the possibility of considering more general "distance"-like functions that can characterize "closeness" between pairs, triples, or larger tuples of elements.

3. Just like we have higher-order Hilbert spaces, which naturally generalize the concept of a dot-product to multiple arguments, one could envision higher-order metrics, and develop them axiomatically (not just norms induced by appropriate higher-order dot products).

4. A more esoteric example from my own work. For certain strictly convex functions $\phi: X \to \mathbb{R}$, the difference \begin{equation*} s(x,y) := \frac{\phi(x)+\phi(y)}{2}-\phi\left(\frac{x+y}{2}\right) \end{equation*} is the square of a metric, which in lucky cases might in fact be a Hilbertian metric. This suggests that a limited amount of generalization is offered by considering \begin{equation*} s_n(x_1,x_2,\ldots,x_m) := \frac{\phi(x_1)+\cdots+\phi(x_m)}{m}-\phi\left(\frac{x_1+\cdots x_m}{m}\right). \end{equation*} This comment is an example of Point 1 above.

Answer 3

There are quite a few people who have tried to generalize metrics to more than two variables. I once tried to track down all the references on this subject for a paper. Here are some:

There is an extensive literature on 2-metrics, in which $d$ takes 3 arguments. This appears to have been introduced by Gahler. Here is a recent example with some references.

What James Cranch mentions in his answer is (I think) originally due to Menger (K. Menger, Untersuchungen uber allgemeine Metrik, Math. Ann. 100.). Menger takes $d$ to be the volume of an $n$-simplex in Euclidean space. Then he tries to abstract away from that. (I can't read German so take this summary with a grain of salt.)

Three recent papers that seek such generalizations are by Deza and Rosenberg (Small cones of $m$-hemimetrics), by Chepoi and Fichet (A note on three-way dissimilarities and their relationship with two-way dissimilarities), and by Warren ($n$-way metrics).

My impression from all of these is that there is no one natural way to extend metrics to take more than 2 arguments.