This was originally a comment on Kristal Cantwell's answer; it is now a complete solution of its own.

As Kristal and others have pointed out, we can assume that the vertices of the pentagon are, in cyclic order, $(1,0)$, $(0,0)$, $(0,1)$, $(x\_1, y\_1)$, $(x\_2, y\_2)$. As she describes, we compute

$$\mathrm{Area}(P) = (1/2) \left( x\_1 + [x\_2 y\_1 - x\_1 y\_2] + y_2 \right)$$

and

$$\mathrm{Area}(Q) = (1/8) \left( x\_1 + [(x\_1+x\_2)(1+y\_1) - x\_1(y\_1 + y\_2)] \right.$$
$$\phantom{\mathrm{Area}(Q)=} \left.+[(1+x\_2)(y\_1+y\_2) - y\_2 (x\_1 + x\_2)] + y\_2 -1 \right).$$

This simplifies to
$$\mathrm{Area}(Q) = (1/8) \left( 2 x\_1 + x\_2 + y\_1 + 2 y\_2 + 2 [ x\_2 y\_1 - x\_1 y\_2] -1 \right).$$

In a previous version of her comment; Kristal and I had different formulas at this point, and I had some discussion of that here. We now seem to both agree on the formulas above.

We want to show that
$$(3/4) \mathrm{Area}(P) - \mathrm{Area}(Q) \geq 0$$.

Plugging in the above formulas, clearing out the $8$ in the denominator, and simplifiying, we want to show that
$$x\_1 - x\_2 - y\_1 + y\_2 + [x\_2 y\_1 - x\_1 y\_2] +1 \geq 0.$$
Rearranging, the left hand side is
$$[(x\_2 -1)(y\_1 -1) - (x\_1-1)(y\_2-1)] + 1$$

The quantity in square brackets is twice the signed area of the triangle $(x\_1, y\_1)$, $(1,1)$, $(x\_2, y\_2)$. (Signed area means positive if the triangle is oriented counter-clockwise, negative if clockwise.) If this is positive, we are done. By the convexity hypothesis, the triangle $(x\_1, y\_1)$, $(0,0)$, $(x\_2, y\_2)$ is positively oriented. So the only way for the term in square brackets to be negative is if $(0,0)$ and $(1,1)$ are on opposite sides of the line $(x\_1, y\_1)$, $(x\_2, y\_2)$. Assume from now on that this is the case.

If we slide $(x\_1, y\_1)$ and $(x\_2, y\_2)$ apart, keeping them on the same line, then the oriented area of $(x\_1, y\_1)$, $(1,1)$, $(x\_2, y\_2)$ will only grow more negative. So we may assume that we have slid them as far apart as possible. That is to say, that $x\_1 = 0$ and $y \_2 =0$. Since $x\_1+y\_1$ and $x\_2 + y\_2$ are $\geq 1$, this means that $x\_2$, $y\_1 \geq 1$.

Then
$$[(x\_2 -1)(y\_1 -1) - (x\_1-1)(y\_2-1)] + 1 \geq$$
$$ 0 \cdot 0 - (-1)(-1) + 1 =0.$$

toodogmatic/hasty about depth $\endgroup$ – Yemon Choi Nov 8 '09 at 8:12