Polynomial roots and convexity A couple of years ago, I came up with the following question, to which I have no answer to this day. I have asked a few people about this, most of my teachers and some friends, but no one had ever heard of the question before, and no one knew the answer.
I hope this is an original question, but seeing how natural it is, I doubt this is the first time someone has asked it.
First, some motivation. Take $P$ any nonzero complex polynomial. It is an easy and classical exercise to show that the roots of its derivative $P'$ lie in the convex hull of its own roots (I know this as the Gauss-Lucas property). To show this, you simply write $P = a \cdot \prod_{i=1}^{r}(X-\alpha_i)^{m_i}$ where the $\alpha_i~(i=1,\dots,r)$ are the different roots of $P$, and $m_i$ the corresponding multiplicities, and evaluate $\frac{P'}{P}=\sum_i \frac{m_i}{X-\alpha_i}$ on a root $\beta$ of $P'$ which is not also a root of $P$. You'll end up with an expression of $\beta$ as a convex combination of $\alpha_1,\dots,\alpha_r$. It is worth mentioning that all the convex coefficients are $>0$, so the new root cannot lie on the edge of the convex hull of $P$'s roots.
Now fix $P$ a certain nonzero complex polynomial, and consider $\Pi$, its primitive (antiderivative) that vanishes at $0:~\Pi(0)=0$ and $\Pi'=P$. For each complex $\omega$, write $\Pi_{\omega}=\Pi-\omega$, so that you get all the primitives of $P$. Also, define for any polynomial $Q$, $\mathrm{Conv}(Q)$, the convex hull of $Q$'s roots.
MAIN QUESTION: describe
$\mathrm{Hull}(P)=\bigcap_{\omega\in\mathbb{C}}\mathrm{Conv}(\Pi_{\omega})$.
By the property cited above, $\mathrm{Hull}(P)$ is a convex compact subset of the complex plane that contains $\mathrm{Conv}(P)$, but I strongly suspect that it is in general larger.
Here are some easy observations:


*

*replacing $P$ (resp. $\Pi$) by $\lambda P$ (resp. $\lambda \Pi$) will not change the result, and considering $P(aX+b)$ will change $\mathrm{Hull}(P)$ accordingly. Hence we can suppose both $P$ and $\Pi$ to be monic. The fact that $\Pi$ is no longer a primitive of $P$ is of no consequence.

*the intersection defining $\mathrm{Hull}(P)$ can be taken for $\omega$ ranging in a compact subset of $\mathbb{C}$: as $|\omega| \rightarrow \infty$, the roots of $\Pi_{\omega}$ will tend to become close to the $(\deg (P)+1)$-th roots of $\omega$, so for large enough $\omega$, their convex hull will always contain, say, $\mathrm{Conv}(\Pi)$.

*$\mathrm{Hull}(P)$ can be explicitly calculated in the following cases:
$P=X^n$, $P$ of degree $1$ or $2$. There are only 2 kinds of degree $2$ polynomials: two simple roots or a double root. Using $z\rightarrow az+b$, one only has to consider $P=X^2$ and $P=X(X-1)$. The first one yields {$0$}, which equals $\mathrm{Conv}(X^2)$, the second one gives $[0,1]=\mathrm{Conv}(X(X-1))$.
Also, if $\Pi$ is a real polynomial of odd degree $n+1$ that has all its roots real and simple, say $\lambda_1 < \mu_1 < \lambda_2 < \dots < \mu_n < \lambda_{n+1}$, where I have also placed $P$'s roots $\mu_1, \dots, \mu_n$, and if you further assume that $\Pi(\mu_{2j}) \leq \Pi(\mu_n) \leq\Pi(\mu_1) \leq\Pi(\mu_{2j+1})$ for all suitable $j$ (a condition that is best understood with a picture), then $\mathrm{Hull}(P)=\mathrm{Conv}(P)=[\mu_1,\mu_n]$: just vary $\omega$ between $[\Pi(\mu_n), \Pi(\mu_1)]$; the resulting polynomial $\Pi_{\omega}$ is always split over the real numbers and you get 
$$[\mu_1,\mu_n]=\mathrm{Conv}(P)\subset\mathrm{Hull}(P)\subset
\mathrm{Conv}(\Pi_{\Pi(\mu_1)})\cap
\mathrm{Conv}(\Pi_{\Pi(\mu_n)}) = \\= [\mu_1,\dots]\cap
[\dots,\mu_n]=[\mu_1,\mu_n]$$


*The equation $\Pi_{\omega}(z)=\Pi(z)-\omega=0$ defines a Riemann surface, but I
don't see how that could be of any use.


Computing $\mathrm{Hull}(P)$ for the next most simple
polynomial $P=X^3-1$ has proven a challenge, and I can only conjecture what
it might be.
Computing $\mathrm{Hull}(X^3-1)$ requires factorizing degree 4
polynomials, so one naturally tries to look for good values of $\omega$,
the $\omega$ that allow for easy factorization of $\Pi_{\omega}=X^4-4X-\omega$---for instance, the $\omega$ that produces a double root. All that remains to be done
afterwards is to factor a quadratic polynomial. The problem is symmetric,
and you can focus on the case where 1 is the double root (i.e., $\omega=-3$).
Plugging in the result in the intersection, and rotating twice, you obtain the following superset of
$\mathrm{Hull}(X^3-1)$: a hexagon that is the intersection of three similar isoceles
triangles with their main vertex located on the three third roots of unity $1,j,j^2$
QUESTION: is this hexagon equal to $\mathrm{Hull}(X^3-1)$?
Here's why I think this might be.
Consider the question of how the convex hulls of the roots of $\Pi_{\omega}$ vary as $\omega$ varies.
When $\omega_0$ is such that all roots of $\Pi_{\omega_0}$ are simple, then the inverse function theorem shows that the roots of $\Pi_{\omega}$
with $\omega$ in a small neighborhood of $\omega_0$ vary holomorphically $\sim$
linearly in $\omega-\omega_0$: $z(\omega)-z(\omega_0)\sim \omega-\omega_0$. If however $\omega_0$ is such that $\Pi_{\omega_0}$
has a multiple root $z_0$ of multiplicity $m>1$, then a small variation of $\omega$
about $\omega_0$ will split the multiple root $z_0$ into
$m$ distinct roots of $\Pi_{\omega}$ that will spread out roughly as
$z_0+c(\omega-\omega_0)^{\frac{1}{m}}$, where $c$ is some nonzero coefficient. This
means that for small variations, these roots will move at much higher velocities
than the simple roots, and they will constitute the  major contribution to the variation of
$\mathrm{Conv}(\Pi_{\omega})$; also, they spread out evenly, and (at least if the
multiplicity is greater or equal to $3$) they will tend to increase the convex hull
around $z_0$. Thus it seems not too unreasonable to conjecture that the convex hull
$\mathrm{Conv}(\Pi_{\omega})$ has what one can only describe as
critical points at the $\omega_0$ that produce roots with
multiplicities. I'm fairly certain there is a sort of calculus on convex sets that
would allow one to make this statement precise, but I don't know see what it could be.
Back to $X^3-1$: explicit calculations suggest that up to second order, the double root $1$ of $X^4-4X+3-h$ for $|h|<<1$ splits in half nicely (here $\omega=-3+h$), and the convex hull will continue to contain the aforementioned hexagon. 
QUESTION (Conjecture): is it true that
$\mathrm{Hull}(P)=\bigcap_{\omega\in\mathrm{MR}}\mathrm{Conv}(\Pi_{\omega})$, where
$\mathrm{MR}$ is the set of all $\omega_0$ such that
$\Pi_{\omega_0}$ has a multiple root, i.e., the set of all $\Pi(\alpha_i)$ where the
$\alpha_i$ are the roots of $P$?
All previous examples of calculations agree with this, and I have tried as best I can to justify this guess heuristically.
Are you aware of a solution? Is this a classical problem? Is anybody brave enough to
make a computer program that would compute some intersections of convex hulls
obtained from the roots to see if my conjecture is valid?
 A: We can characterize those $P$ for which $\mathrm{Hull}(P) = \mathrm{Conv}(P)$. 
First suppose that the roots of $P$ do not lie on a line. We prove that $\mathrm{Hull}(P) = \mathrm{Conv}(P)$ if and only if there is an antiderivative $Q$ of $P$ for which $Q = A^2B$, and the roots of $B$ lie in the convex hull of $A$. The "if" is immediate and left to the reader. 
Note that a root $\beta$ of $P$ can lie on the boundary of $\mathrm{Conv}(\Pi_\omega)$ only if $\beta$ is a root of $\Pi_\omega$, or all the roots of $\Pi_\omega$ lie on a line. This latter case of course is impossible if the roots of $P$ do not lie on a line.  
Let $\beta$ be a root of $P$. We claim that for every neighborhood $N$ of $\Pi(\beta)$ there is a neighborhood $M$ of $\beta$ such that $\mathrm{Conv}(\Pi_y) \supset M$ for every $y \in \mathbb C \setminus N$. This certainly holds for any given $M$ when $y$ lies outside a large compact subset of $\mathbb C$, so we can think of $y$ ranging over a compact set. For each $y \neq \Pi(\beta)$, there is a ball of positive radius around $\beta$ in $\mathrm{Conv}(\Pi_y)$, and the size of the maximal such ball varies continuously, so the claim follows. 
Now, suppose that $\beta$ and $\gamma$ are adjacent vertices (extreme points) of $\mathrm{Conv}(P)$. Suppose $\gamma$ is not a root of $\Pi_{\Pi(\beta)}$. Then $\gamma$ lies in the interior of $\mathrm{Conv}(\Pi_{\Pi(\beta)})$, and we can find $\gamma'$ in the interior of $\mathrm{Conv}(\Pi_{\Pi(\beta)})$ so that $\overline{\gamma' \beta} \cap \mathrm{Conv}(P) = \beta$. Then $\overline{\gamma' \beta} \subset \mathrm{Conv}(\Pi_y)$ for all $y$ in a suitable neighborhood $N$ of $\Pi(\beta)$. On the other hand, $M \subset \mathrm{Conv}(\Pi_y)$ for a suitable neighborhood $M$ of $\beta$ and all $y \notin N$. Therefore $\overline{\gamma' \beta} \cap M \subset \mathrm{Hull}(P)$, 
and hence $\mathrm{Conv}(P) \subsetneq \mathrm{Hull}(P)$.
So if $\mathrm{Conv}(P) = \mathrm{Hull}(P)$, then $\Pi(\beta) = \Pi(\gamma)$ for all adjacent vertices $\beta, \gamma$ of $\mathrm{Conv}(P)$. Letting $Q$ be $\Pi(\beta)$ for any extreme point $\beta$ of $\mathrm{Conv}(P)$, we see that every extreme point of $\mathrm{Conv}(P)$ is a root of $Q$, and hence a double root of $Q$. Moveover, if $\mathrm{Conv}(P) = \mathrm{Hull}(P)$, then $\mathrm{Conv}(P) \supseteq \mathrm{Conv}(Q)$, so $Q = A^2B$, where the roots of $B$ lie in the convex hull of the roots of $A$, the extreme points of $\mathrm{Conv}(P)$.
Now suppose that the roots of $P$ lie on a line. We can assume that $P$ is real (with positive leading term) and all of the roots of $P$ are real as well. Let $\beta$ and $\gamma$ be the least and greatest root of $P$, respectively. Then, by a similar analysis, $\mathrm{Hull}(P) = \mathrm{Conv}(P)$ if and only if


*

*$\Pi_y$ has all real roots for some value of $y$,

*The roots of $\Pi_{\Pi(\beta)}$ have real part at least $\beta$, and

*The roots of $\Pi_{\Pi(\gamma)}$ have real part at most $\gamma$. 


You can easily verify that 1, 2, and 3 hold when $P$ has degree 3. I would be tempted to conjecture that 2 and 3 always hold (when the roots of $P$ are real).
A: First, a counterexample to your conjecture. Let $\Pi = x^4+x^3+4x^2+4x = x(x+1)(x^2+4)$, so $P = 4x^3+3x^2+8x+4$. The critical values of $\omega$ are $1.06638, 3.89455 + 2.87687i, 3.89455 - 2.87687i$, and by inspection (using Mathematica) we see that for each of these values of $\omega$, $\mbox{Conv}(\Pi_\omega)$ contains a neighborhood of $0$.
Now for a calculus on convex sets. Every convex set is the intersection of a set of halfplanes. Call a halfplane in this collection essential if removing all of the halfplanes in an open set of halfplanes (in the halfplane topology) containing it from our set of halfplanes makes the intersection of the halfplanes in our set bigger. We wish to find a characterization of the essential halfplanes of $\mbox{Hull}(P)$.
First of all, I claim that any essential halfplane of $\mbox{Hull}(P)$ occurs as an essential halfplane of $\mbox{Conv}(\Pi_\omega)$ for some $\omega$. This follows from continuity - for any open set around our essential halfplane there is some $\omega$, take the limit of a subsequence of these $\omega$s...
Now, suppose that the halfplane $\mbox{Re}(x) \le 0$ occurs as an essential halfplane of some $\mbox{Conv}(\Pi_\omega)$, i.e. there are at least two roots of $\Pi_\omega$ with real part $0$, and the rest of the roots have negative real part. If the number of roots on the line $\mbox{Re}(x) = 0$ (counted with multiplicity) is two, then by holomorphicity we can always find a direction to move $\omega$ so that either both roots move to the left, or both roots stay on the line $\mbox{Re}(x) = 0$ and move towards eachother. If we can ever make both roots move to the left, then clearly the halfplane $\mbox{Re}(x) \le 0$ is not an essential halfplane of $\mbox{Hull}(P)$, otherwise we keep pushing the roots towards eachother until either they run into eachother or until a third root hits the line $\mbox{Re}(x) = 0$. In any case, we see that if a halfplane is essential for $\mbox{Hull}(P)$, then there is some $\omega$ such that the halfplane is essential for $\mbox{Conv}(\Pi_\omega)$ and such that at least three roots (counted with multiplicity) of $\Pi_\omega$ are on the boundary of the halfplane, or two of the roots are equal and $\Pi_\omega$ has no other roots.
So if we let $L$ be the set of $\omega$s such that three of the roots of $\Pi_\omega$ lie on a line, we get that $\mbox{Hull}(P) = \cap_{\omega \in L} \mbox{Conv}(\Pi_\omega)$ if $\deg P \ge 2$.
Edit: Actually, I think there is a problem with this. It's conceivable that two roots are on the line $\mbox{Re}(x) = 0$ and have derivatives (with respect to $\omega$) pointed in opposite directions, such that we can't simply push them towards eachother. For instance, the map from one root to the other root could, locally, look like the fractional linear transform sending the left halfplane to a circle contained in the right halfplane and tangent to the line $\mbox{Re}(x) = 0$ at the other root. So, we may need to enlarge the set $L$ to contain also those $\omega$s for which the ratio of the derivatives of two of the roots (with respect to $\omega$) is a negative real number.
Edit 2: It turns out that this isn't an issue. Call the two roots on the line $\mbox{Re}(x) = 0$ $r_1$ and $r_2$. Suppose that locally, $r_1(\epsilon) = \epsilon$, $r_2(\epsilon) = i - m\epsilon + a\epsilon^k + O(\epsilon^{k+1})$, $a \ne 0$, $m > 0$. Note that if we had $r_2(\epsilon) = i-m\epsilon$, then the intersection of the halfplanes corresponding to $r_1(\epsilon), r_2(\epsilon)$ and $r_1(-\epsilon), r_2(-\epsilon)$ would be contained  in the halfplane $\mbox{Re}(x) \le 0$, and the intersection of their boundaries would be located at $i/(m+1)$. Now if $k$ is even, then the correction term shifts the intersection of the boundaries by $a\epsilon^k/(m+1) + O(\epsilon^{k+1})$, so if we choose $\epsilon$ small such that $a\epsilon^k$ is real and negative, then we see that the halfplane $\mbox{Re}(x) \le 0$ is not essential. If $k$ is odd, then if we choose $\epsilon$ small such that $\mbox{Re}(\epsilon) < 0$ and $a\epsilon^k$ is a positive real times $i$, then $r_2(\epsilon)$ is shifted up and $r_2(-\epsilon)$ is shifted down, so the intersection of the boundaries will be shifted to the left (draw a picture), so again the halfplane $\mbox{Re}(x) \le 0$ is not essential.
A: I. First I want to share some computer experiments of H.H. Rugh. The following image supports the positive answer of the 
QUESTION: is this hexagon equal to $Hull(X^3−1)$?
triangle (as a new user I was not allowed to use image tags).
A scilab program  testing this problem can be found at 
roots-dancing. In particular an example $z^6-3z^3+z$ similar to the starting
example of Zeb, showing that $Hull(P)$ can be strictly smaller then $\bigcap_vConv(f_v)$
for $v$ ranging the critical values of $f$ (shown in blue polygons).
See smaller.

II. Here is a proof (communication of Rugh) of the same statement of Zeb (with $f=\Pi$):
If we let L be the set of $v$'s such that three of the roots of $f_v$ lie on a line, we get $Hull(P)=\bigcap_{v\in L}Conv(f_v)$ if $deg f≥3$.
The underlying idea is very similar to that of Zeb.
Statements: Let $f$ be a polynomial of degree at least 3. Assume that $a_0$ and $b_0$ are two distinct simple roots of $f(z)-v_0$. Then for $v$ in a small neighborhood of $v_0$, there are two simple roots
 $a(v)$, $b(v)$ of $f(z)-v$ with $a(v_0)=a_0$ and $b(v_0)=b_0$. In this case,


*

*If for some fixed complex number $t\neq 0,1$,
the  holomorphic function $v\mapsto t a(v) + (1-t)b(v)$ is a constant $c$, then $c$ is a critical point of $f$ and $f$ has a rotational symmetry about $c$.

*If the segment $[a_0, b_0]$ is a boundary  edge  of the polygon $Conv(f_{v_0})$ and no other point in the line through $a_0, b_0$ is
mapped to $v_0$, then 
2.1 for $v$ sufficiently close to $v_0$, the segment $[a(v),b(v)]$ is a boundary edge of the polygon $Conv(f_{v})$,
and for any $t\in ]0,1[$, the 
map $v\mapsto ta(v)+(1-t) b(v)$ is an open mapping.
2.2. The line through $a_0, b_0$ is outside $\bigcap_v Conv(f_{v})$.
Proof.
 1. Replacing $f(z)$ by $f(z-c)$ if necessary, we may assume $c=0$.
 Note that $a\mapsto b(f(a))$ is defined and holomorphic in a neighborhood of $a_0$, satisfying that $b(f(a_0))=b_0$ and $f(b(f(a)))=f(a)$.
 It follows that $ta+(1-t)b(f(a))\equiv 0$ so $b(f(a))=\dfrac{ta}{t-1} $.
 Therefore $f(a)=f(\dfrac {ta}{t-1})$ in a neighborhood of $a$, thus in the entire complex plane. 
 Comparing the coefficients we conclude that $\dfrac t{t-1}$ is a root of unity and $f'(0)=\dfrac{t}{t-1} f'(0)$.
Using $\dfrac t{t-1}\ne 1$ we get $f'(0)=0$.  An example is $z^6+3z^4-5z^2$. 
2.1. The condition means that all the other points in $f^{-1}(v_0)$ are contained in one of the open half planes delimited  by the line through $a_0, b_0$. This
is clearly an open condition.
Now if $ ta(v)+(1-t) b(v)\equiv c$, by Point 1 $c$ must be a critical point and a center of symmetry and 
 $Conv(f_{v_0})$ would have been symmetric with respect to $c$. This is not possible by our assumption that all the other
 points of $f^{-1}(v_0)$ (and there is at least one due to the assumption
on the degree of $f$ and on the simplicity of $a_0, b_0$ as roots of $f_{v_0}$) are on one side of the line through $a_0, b_0$. 
2.2. We may look at the open set $W=\bigcup_v Outer(f_v)$ where $Outer(f_v)$ is the complement of $Conv(f_v)$. 
 We may assume $a_0, b_0$ are on the imaginary axis and all the other points of $f^{-1}(v_0)$ are
 on the left half plane.  We know already $i{\mathbb R}-[a_0, b_0]\subset Outer(f_{v_0})\subset W$.
Fix some $t\in [0,1]$ and let $z_0=ta_0+(1-t)b_0$. We may assume $z_0=0$.  Now $v\mapsto z(v)=t a(v)+(1-t) b(v)$ is
 open. Choose a path $v(s)$ such that $z(v(s))$ is negative real. Then
the segment $[a(v(s)), b(v(s))]$ passes through $z(v(s))$ and remains almost vertical for sufficiently small $s$, so has $z_0$ on its right side. By 2.1 we may conclude
 $z_0\in Outer(f_{v(s)})\subset W$. qed.

III. Finally I want to share some numerical experiments (with the help of Jos Leys)
illustrating a refinement (communication by Thurston) of Gauss-Lucas property.
Consider a polynomial $f$ as a branched covering of the complex plane. Denote by ${\cal C}$ the convex hull of the critical points of $f$. It is called {\em the critical convex} of $f$. The following statements are equivalent:
(1) For any $v\in \mathbb C$, we have $Conv(f_v)\supset {\cal C}$.
(2) The map $f$ is surjective on any closed half plane $H$ intersecting ${\cal C}$. 
(1)$\Longrightarrow$(2). Assume $f(H)\not\ni v$. Then $f^{-1}(v)$ is contained in $\mathbb C-H$ which is an open half plane, in particular convex.
Then $Conv(f|_v)$ is also contained in $\mathbb C - H$. So $Conv(f|_v)\not\supset {\cal C}$, contradicting (1).
(2)$\Longrightarrow$(1). Assume that $Conv(f|_v)$ does not contain the entire set ${\cal C}$. Then there is a 
closed half plane $H$ intersecting ${\cal C}$ but disjoint from $Conv(f|_v)$. Then $f(H)\not\ni v$, that is, $f$ is not surjective on $H$, contradicting (2).
Now the refinement (I'll leave the proof to Thurston if somebody requires) is that if one takes a supporting line $L$ of $\cal C$
there is a region on the outer half space of $L$ on which $f$ is a bijection onto $\mathbb C$ (injective in the interior and bijective on the union of the interior with
half of the boundary arc).
In fact this region is bounded by two geodesic rays of the conformal metric $|f'(z)|\cdot |dz|$
tangent to $L$ at a critical point (if the critical point is simple, otherwise
the region is even smaller).  This means that we use the euclidean length in the range to measure tangent vectors in the domain. Geodesics in this metric are the pullbacks by $f$ of straight lines.
The movie bijectivity is made by Jos Leys to illustrate this result.
A: This problem has been considered before:
The notes to chapter 4 of 
Rahman/Schmeißer: Analytic Theory of Polynomials, Oxford University Press, 2002
mention this problem, state that conv(P) is a proper subset of hull(P) in general, and give two references:
1) J. L. Walsh: The location of Critical Points of Analytic and Harmonic Functions,
AMS Colloquium Publications Volume 34, 1950, p. 72 
2) E. Chamberlin and J. Wolfe: Note on a converse of Lucas's theorem, Proceedings of the American Mathematical Society 5, 1954, pp. 203 - 205
I had a look at both of them, the paper of Chamberlin and Wolfe can be obtained online via the AMS for free. 
The relevant paragraph in Walsh's book is 3.5.1 (starts on page 71). I state the 4 theorems given there for convenience:
1) conv(P) = hull(P) if P is of degree 1 or 2
2) conv(P) = hull(P) if P is of degree 3 and its zeros are collinear
3) There exists a P of degree 3, such that conv(P) is a proper subset of hull(P); example $P(z) = z^3 + 1$.
4) There exists a P of degree 4 with real zeros, such that conv(P) is a proper subset of hull(P); example $P(z) = (z^2 - 1)^2$.
Chamberlin and Wolfe prove, that 
1) Any point on the boundary of hull(P) is on the boundary of conv($\Pi_\omega$) for some $\omega$.
2) If a side of any conv($\Pi_\omega$) contains a point of hull(P) and only two zeros of $\Pi_\omega$ , counting multiplicities, then P is of degree 1. (Here a side is equal to conv($\Pi_\omega$), if the zeros of $\Pi_\omega$ are collinear.)
3) Vertices of conv(P) need not lie on the boundary of hull(P), example
    $\Pi(z) = z^2 (z + 1)(z^2 - 2az + 1 + a^2)$, where a is positive and sufficiently small.
4) Even if P is cubic, hull(P) need not be determined by its primitives with multiple zeros, example P(z) = $4z^3 + 9/2 z^2 + 2z + 3/2$.
While I have corrected what I considered a minor typo in the example $\Pi(z) = z^2 (z + 1)(z^2 - 2az + 1 + a^2)$, I did no thorough proofreading.
Walsh gives no special references to further work in section 3.5.1 and the only reference provided by Chamberlin and Wolfe is to Walsh's book cited above. 
