Take the 2-minute tour ×
MathOverflow is a question and answer site for professional mathematicians. It's 100% free, no registration required.

A measure of dependence is a way to assign a number (usually normalized between 0 and 1) to a couple of random variable, such that $\delta(X,Y)=0$ if and only of $X$ and $Y$ are independent, and $\delta(X,Y)=1$ as soon as there is a complete dependence between $X$ and $Y$. Rényi introduced in the paper On measures of dependence (Acta Math. Acad. Sci. Hungar. 10 1959 441--451) a set of axioms that a measure of dependence should fulfill. Among them are nonparametricity ($\delta(f(X),g(Y))=\delta(X,Y)$ as soon as $f,g$ are inversible bimeasurable functions, so that $\delta$ dos not rely on a metric, or affine structure unlike the correlation) and symmetry ($\delta(X,Y)=\delta(Y,X)$), and the "complete dependence" is defined by the existence of a relation $X=f(Y)$ of $Y=f(X)$.

I feel that the symmetry assumption is unnatural. In view of, for example, Bell inequalities, one would want to have some sort of triangle inequality, so that if $\delta(X,Y)$ and $\delta(Y,Z)$ are both very high, $\delta(X,Z)$ should be quite high too. This rules out symmetry since one can easily construct random variables such that $X=f(Y)$, $Z=g(Y)$ but $X$ and $Z$ are independent (in this example, one would want to say that $X$ depends heavily on $Y$, but that $Y$ does not depend that much on $X$).

Question: do you know a measure of dependence, or a similar tool, that is nonparametric and nonsymmetric? Does it satisfy a triangle inequality? Any reference would be useful.

share|improve this question
According to the work of Schweitzer and Wolff, it was not the symmetry condition which is unnatural, rather the "existence of a relation". His original conditions were satisfied only by the maximal correlation coefficient. projecteuclid.org/… –  Gjergji Zaimi Apr 23 '10 at 21:03
@Gjergji Zaimi: this reference is interesting, but I would not call a measure of dependency nonparameteric when it relies so heavily on the order structure (although it is certainly very weakly parametric, since the order is quite flexible compared to an affine structure). Imagine for example that you have a sociological study and you want to examine the correlation between the gender and some other characteristic, you do not have a structure, not even an order, on the set of gender. –  Benoît Kloeckner Apr 24 '10 at 6:32

2 Answers 2

Conditional entropy might be close to what you are looking for. You will have to transform it in an obvious way to fit Rényi's framework. See http://en.wikipedia.org/wiki/Conditional_entropy and especially the section titled "Intuition".

share|improve this answer
This is effectively close to what I am looking for; my question could mostly be recast into: « has (possibly renormalized) conditional entropy be used has a measure of dependency, and can it be nonparametrically extended to non-discrete random variable? » –  Benoît Kloeckner Apr 26 '10 at 8:21
Conditional entropy is often used as a measure of dependency: one specific example is in the realm of databases, where it's used to capture weak functional dependencies (Dalkilic/Robertson, Principles Of Database Systems, 2000). There, it's discrete. In general, you can define it via an integration, but like with differential entropy, you have to be careful –  Suresh Venkat Apr 27 '10 at 0:59

Please check my recent work "On nonsymmetric nonparametric measure of dependence" at


It may answer your question.

This article describes a new class of copula-based dependence measure $\delta(X,Y)$ which has value one if and only if the variable $Y$ is a function of the other variable $X$, and which is invariant under 1-1 transformation of $X$. This class is nonsymmetric, and the dependence measure of $X$ on $Y$ can be small even if the dependence measure of $Y$ on $X$ is one.

share|improve this answer
It seems interesting, but could you give here a brief account of what you achieve in this paper? –  Benoît Kloeckner yesterday
It describes a new class of copula-based dependence measure which has value one if and only if one variable Y is a function of the other variable X, and is invariant under 1-1 transformation of X. Symmetric measures may equal to one even if the variables X and Y do not have functional dependence. This class is nonsymmetric, so the dependence measure of X on Y can be small even if the dependence measure of Y on X is one. –  Hui Li yesterday
I took the liberty to include the description, slightly edited, to your answer (this is more practical for other readers). Could you please precise which other axioms your measure satisfies? E.g. is the dependency zero if and only if $X$ and $Y$ are independent? Don't you get any kind of invariance under change of parameter on the $Y$ side? –  Benoît Kloeckner 22 hours ago
I am totally new here, so thanks for editing the answer. Yes dependence zero means independence. Due to nonsymmetric nature of the measure, change of parameter on Y may increase or decrease the dependence. For a specific form, the dependence is invariant under monotonic transformation on Y. –  Hui Li 16 hours ago

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.