Suppose $\mu = \mu_{\max}$ is a (Borel) probability measure on $[0,1]$ for which the following integral is maximized: \begin{equation}\tag{1} \mathcal{I}(\mu) := \iiint |(y-x)(z-y)(z-x)| d\mu(x)d\mu(y)d\mu(z) \end{equation} Existence of $\mu_{\max}$ is relatively easy to prove. We will prove the following property:
Proposition: \begin{equation}\tag{2}\label{infinite} \limsup_{\epsilon \to 0} \frac{\mu_{\max}([0,\epsilon])}{\epsilon} = + \infty. \end{equation} That is, $\mu_{\max}$ has infinite density at $0$ and (by symmetry) also at $1$.
Here are some open questions:
- Is $\mu_{\max}$ atomic at $0$ and $1$?
- Is $\mu_{\max}$ absolutely continuous elsewhere?
First, we prove a
Lemma: If $X$ is a random variable with measure $d\mathbb{P}$ and cdf $F(x)$ satisfying $F(x)\le cx$ on $x\in [0,1]$, and $p=F(\epsilon)$, then
\begin{equation} \int_{[X \le \epsilon]} X d\mathbb{P} \ge \frac{p^2}{2c} \, . \end{equation}
Proof of Lemma: By a variant of the well-known identity for a non-negative random variable $X$: \begin{equation} \mathbb{E}(X) = \int_0^\infty (1-F(x))dx, \end{equation} we have \begin{align} \int_{[X\le \epsilon]} X d\mathbb{P} &= \int_0^\epsilon (p-F(x)) dx \\ &\ge \int_0^\epsilon \max(p-cx,0) dx \\ &\ge \int_0^{p/c} (p-cx) dx \\ &= \frac{p^2}{2c}. \end{align} This proves the Lemma.
The proof of the Proposition \eqref{infinite} is by contradiction. Let $\mu=\mu_{\max}$ and let $F(x) := \mu([0,x])$ be the cumulative distribution function. Suppose that \eqref{infinite}If the proposition does not hold, sothen there exists a finite $c \ge 1$ such that, for all $x\in [0,1]$, \begin{equation}\tag{3}\label{linear} F(x) \le cx . \end{equation} Note that $F(x)>0$ for every $x>0$, because if $F(x)=0$ for some $x>0$ then we can increase $I$ by replacing $\mu$ with $h_*\mu$, where $h$ is the affine map that sends $[x,1]$ onto $[0,1]$.
Let $X$, $Y$, $Z$ be independent random variables distributed according to $\mu$; let $\mathbb{P}$ denote the underlying probability. Fix a constant $\gamma \in (0,1)$ such that the following event $\Gamma$ has probability at least $1/2$: \begin{equation}\tag{4}\label{Gamma} \Gamma := \big[ Y>\gamma, \ Z>\gamma, \text{ and } |Y-Z|>\gamma \big]. \end{equation} Consider random variables $X_1 \le X_2 \le X_3$ obtained by ordering $X$, $Y$, $Z$; so $X_1 = \min(X,Y,Z)$, for instance. Fix a number $\epsilon$ in the range \begin{equation}\tag{5}\label{epsilon} 0 < \epsilon < \frac{\gamma^2}{12c}. \end{equation} Define random variables $\tilde{X}_1 \le \tilde{X}_2 \le \tilde{X}_3$ by: \begin{equation}\tag{6} \tilde X_i := \begin{cases} X_i &\text{if } X_i > \epsilon, \\ 0 &\text{otherwise.} \end{cases} \end{equation} Finally, define \begin{align} \Pi &:= (X_2-X_1)(X_3-X_1)(X_3-X_1) , \\ \tilde \Pi &:= (\tilde X_2- \tilde X_1)(\tilde X_3- \tilde X_1)(\tilde X_3- \tilde X_1) . \tag{7} \end{align} We claim that: \begin{equation}\tag{8}\label{mainclaim} \mathbb{E}(\Pi) < \mathbb{E}(\tilde \Pi) . \end{equation} This means that $\mathcal{I}(\mu) < \mathcal{I}(\nu)$, where $\nu$ is the probability measure obtained from $\mu$ by crunching all the mass of the interval $[0,\epsilon]$ on the point $0$. So the claim \eqref{mainclaim} implies that $\mu$ is not a maximizer, and therefore it is sufficient to prove it in order to conclude \eqref{infinite}.
Note that $\tilde \Pi = \Pi$ whenever $X_1 > \epsilon$, and so: \begin{equation}\tag{9} \mathbb{E}(\tilde \Pi - \Pi) = \int_{[X_1 \le \epsilon]} (\tilde \Pi - \Pi) d\mathbb{P} . \end{equation} On the other hand, $X_2 \le \epsilon$ implies $\tilde \Pi = 0$, so: \begin{equation}\tag{10}\label{difference} \mathbb{E}(\tilde \Pi - \Pi) = - \int_{[X_2 \le \epsilon]} \Pi d\mathbb{P} + \int_{[X_1 \le \epsilon < X_2]} (\tilde \Pi - \Pi) d\mathbb{P} =: -I_1 + I_2 . \end{equation} Let us estimate the integrals $I_1$ and $I_2$.
Let \begin{equation}\tag{11}\label{p} p := \mu([0,\epsilon]) = \mathbb{P}[X\le \epsilon]. \end{equation} Then: \begin{align} \mathbb{P}[X_2\le \epsilon] &= \mathbb{P}[X_2\le \epsilon < X_3] + \mathbb{P}[X_3\le \epsilon] \\ &= 3p^2(1-p) + p^3 \\ &\le 3p^2 \, . \tag{12} \end{align} On $[X_2 \le \epsilon]$ we have $\Pi \le X_2-X_1 \le X_2 \le \epsilon$. Therefore we obtain the bound \begin{equation}\tag{13}\label{I1} I_1 \le 3 \epsilon p^2 . \end{equation}
Let us estimate $I_2 = \int_{[X_1 \le \epsilon < X_2]} (\tilde \Pi - \Pi) d\mathbb{P}$ from below. Note that the integrand is non-negative, and so \begin{equation}\tag{14} I_2 \ge \int_{G} (\tilde \Pi - \Pi) d\mathbb{P} \end{equation} for any measurable set $G \subseteq [X_1 \le \epsilon < X_2]$. We choose $G:= [X \le \epsilon] \cap \Gamma$, where $\Gamma$ is as in \eqref{Gamma}. Over $G$ we have: \begin{align} \tilde \Pi - \Pi &= YZ |Y-Z| - (Y-X) (Z-X) |Y-Z| \\ &= X(Y + Z - X) |Y-Z| \\ &\ge \gamma^2 X.\tag{15} \end{align} Therefore: \begin{equation}\tag{16} I_2 \ge \int_{G} (\tilde \Pi - \Pi) d\mathbb{P} \ge \gamma^2 \int_{[X \le \epsilon] \cap \Gamma} X d\mathbb{P} = \gamma^2 \mathbb{P}(\Gamma) \int_{[X \le \epsilon]} X d\mathbb{P} \end{equation} (since the random variable $X$ is independent from the event $\Gamma$).
We will show later that: \begin{equation}\tag{17}\label{claim} \int_{[X \le \epsilon]} X d\mathbb{P} \ge \frac{p^2}{2c} \, . \end{equation} As a consequenceThen by the lemma, $I_2 \ge \frac{\gamma^2 p^2}{4c}$$I_2 \ge \gamma^2 p^2/4c$ and therefore: \begin{equation}\tag{18} \mathbb{E}(\tilde \Pi - \Pi) = I_2 - I_1 \ge \left(\frac{\gamma^2}{4c} - 3 \epsilon \right) p^2 > 0, \end{equation}\begin{equation}\tag{17} \mathbb{E}(\tilde \Pi - \Pi) = I_2 - I_1 \ge \left(\frac{\gamma^2}{4c} - 3 \epsilon \right) p^2 > 0, \end{equation} by our choice of $\epsilon$. This proves the claim \eqref{mainclaim} and the main result, modulo the inequality \eqref{claim}.
Recall the following well-known identity for the expected value of a non-negative random variable $X$: \begin{equation}\tag{19} \mathbb{E}(X) = \int_0^\infty (1-F(x))dx . \end{equation} From this we can deduce that the conditional expectation \begin{equation}\tag{20} \mathbb{E}(X|X\le \epsilon) := \frac{1}{F(\epsilon)} \int_{[X\le \epsilon]} X d\mathbb{P} \end{equation} satisfies the following identity: \begin{equation}\tag{21} \mathbb{E}(X|X\le \epsilon) = \int_0^\epsilon \left( 1 - \frac{F(x)}{F(\epsilon)} \right) dx . \end{equation} Specializing to our case, and using \eqref{p} and \eqref{linear}: \begin{align} \int_{[X\le \epsilon]} X d\mathbb{P} &= \int_0^\epsilon (p-F(x)) dx \\ &\ge \int_0^\epsilon \max(p-cx,0) dx \\ &\ge \int_0^{p/c} (p-cx) dx \\ &= \frac{p^2}{2c},\tag{22} \end{align} proving \eqref{claim}Proposition.