About the ratio of the areas of a convex pentagon and the inner pentagon made by the five diagonals Question : Letting $S{^\prime}$ be the area of the inner pentagon made by the five diagonals of a convex pentagon whose area is $S$, then find the max of $\frac{S^\prime}{S}$.
                  

I've been interested in this simple question. It seems that a regular pentagon and its affine images would give the max, but I'm facing difficulty.
Remark : This question has been asked previously on math.SE without receiving any answers.
 A: Here's a possible strategy:
Consider deformations of a vertex of the convex (outer) pentagon which preserve area. The vertex is constrained to move along a line parallel to the segment connecting the adjacent vertices, and clearly must remain in a compact interval if the pentagon is to remain convex. The area of the inner pentagon may be written as a sum and difference of areas of triangles, each of which has at most one vertex move along a line segment during the deformation, so has area proportional to the length of an edge. The vertices of the inner pentagon are determined by projective transformations of the moving vertex, so have coordinates along the fixed segments given by linear fractional transformations of the displacement. Thus, the area of the pentagon is a sum of rational functions, and it turns out each is concave down on the interval, so the area is a concave rational function with a unique maximum. One should therefore get 5 constraints on the pentagon vertices. 
Up to affine transformations, there are a 4-parameter family of pentagons (e.g. by fixing 3 vertices, as David Speyer suggested in a comment). So one obtains an overdetermined system, which hopefully has a unique (or finitely many) solutions, one of which is the (affine) regular pentagon. 
A: A complete solution is available at https://arxiv.org/abs/1812.07682
On the polygon determined by the short diagonals of a convex polygon,
Jacqueline Cho, Dan Ismailescu, Yiwon Kim, Andrew Woojong Lee
A: Not an answer, just a long comment. This comment slowly evolved into a partial numerical solution: the regular pentagon is the local maximum.
I've used Maple to produce the formula for $\frac{S'}{S}$ and the result is a rational function on $\mathbb{R}^{10}$ with both numerator and denominator of degree 12. The numerator has 11495 monomial terms while the denominator is factored into product of six degree two polynomials. This means that the symmetry of the problem is somehow lost during the computation and so one must be a bit careful when using such programs.
Nevertheless, restricting the pentagon to have the first side on the $x$-axes I get numerator that factorizes as product of $x_2$ and a polynomial with 182 monomials. While analytical solution is still most probably out of the question, it may be possible to use this for numerical experiments. 
For example, working on $\mathbb{R}^{10}$, I was able to verify that the regular pentagon is a stationary point. To my surprise, Maple computed the eigenvalues of the Hessian at this point analytically. Four of the eigenvalues are negative, but unfortunately there are six eigenvalues that are numerically zero ($\approx 10^{-9}$).
I think it could be possible to "prove" this way that the regular pentagon is a local maximum. I'll try to look at it again tomorrow.

Edit: Let me summarize the comments so far.
As David Speyer mentioned, all pentagons are affinely congruent to those whose vertices lies on a conic. If one uses the rational parametrization of the circle $t \to \left(\frac{t}{1+t^2}, \frac{1-t^2}{1+t^2}\right)$, then the degree of the resulting polynomials in the Jacobian rise to more than 30. Nevertheless, one can factor out terms like $t$ or $t^2 + 1$. Of course, we can still fix one vertex and surprisingly, it matters (at least in Maple) which one. Fixing $(0,1)$ as a vertex leads to better results with degrees $(29,29,29,29)$ and number of monomials $(2348,2754,2754,2348)$. This is still unsolvable (at least in Maple on my computer). But weep not for we can at least compute the Hessian at the regular pentagon and get it's eigenvalues  (in this case only numerically, analytic computation timed out): $-0.174, -0.075, -0.001, -0.003$. 
We can conclude (if we believe Maple) that the regular pentagon is the local maximum among "circle pentagons".
Dag Oskar Madsen reminded me that I didn't use all the freedom we have in the problem. By fixing  $(0,0), (1,0)$ and $(0,1)$ I obtained rational function in four variables with numerator of degree 8 with 53 terms and denominator of degree 10 factored into a product of 6 terms. The numerators of the Jacobian have the following structure: "Degrees:", [16, 16, 15, 15], "Terms:", [637, 633, 424, 510] from which we see that this resulted in smallest system so far. On the other hand, any symmetry in the equations was most probably destroyed and thus it may be harder to solve it. 
Fixing vertices $1,2,3$ of the regular pentagon with edge $12$ on the $x$-axis I verified that the eigenvalues of the Hessian are indeed negative, namely $-0.027, -0.012, -0.148, -0.136$. This establishes numerically that the regular pentagon is a local maximum in the set of all pentagons.
I've tried to play with reformulation by Noam D. Elkies and treat it like a constrained maximization problem. That is
$$
S = \frac{1}{2}\left(a + \sqrt{a^2 - 4b}\right) \\
S' = S - a + \sum_{i \text{ mod } 5} \frac{A_{i-2}A_{i+2}}{S-A_i},\\
\text{where}\\
a = \sum_{i \text{ mod } 5} A_i, \quad b = \sum_{i \text{ mod } 5} A_iA_{i+1}
$$
and $A_i$ is the area of triangle formed by three consecutive vertices $P_{i-1}P_iP_{i+1}$. The problem is to maximize $S'$ relative to $S=1$.
The system for Lagrange multipliers reduces to  5 rational equations with numerators of degree  12 with 1544 terms
$$
\frac{\partial L}{\partial a_i} = f_i + \left(\lambda + \sum_{j \text{ mod } 5} \frac{A_{j-2}A_{j+2}}{(1-A_j)^2}  \right)g_i =0,
$$
where
$$
g_i = \frac{1}{2}\left( 1 + \frac{A_i - 4A_{i-1} - 4A_{i+1}}{2-a}\right)\\
f_i = g_i -1 + \sum_{j \text{ mod } 5} \frac{ (A_{j-2}\delta_{i,j+2} + A_{j+2}\delta_{i,j-2})(1-A_j) - A_{j-2}A_{j+2}(g_i - \delta_{i,j})  }{(1-A_j)^2};
$$
plus the constraint 
$$
a - b = 1,
$$
which is quadratic with eleven terms.
I encourage everybody to try this out in their CAS of choice. I got curious about the current state of the art in Groebner basis and asked another question.
A: [Corrected (two typos noted by V.T.) and expanded
(alternative coordinates)]
Also not an answer, but this time a simpler algebraic formulation.
All indices are cyclic mod $5$.  Let the vertices be $P_i$ ($i \bmod 5$),
and let $A_i$ be the area of triangle $P_{i-1} P_i P_{i+1}$.
Then $S$ is the larger root of the quadratic equation
$$
S^2 - S \sum_{i\,\bmod\,5} A_i + \sum_{i\,\bmod\,5} A_i A_{i+1},
$$
and
$$
S' = S - \sum_{i\,\bmod\,5} A_i + \sum_{i\,\bmod\,5} \frac{A_{i-2} A_{i+2}}{S-A_i}.
$$
So we are to prove that as $A_i$ vary over all positive numbers
the ratio $S'/S$ is maximized when the $A_i$ are all equal.
The formula for $S$ was obtained by choosing coordinates so that
the $P_i$ are at $(0,0)$, $(0,1)$, $(1,0)$, $(x_2,y_2)$, $(x_3,y_3)$
(as suggested by Dag Oskar Madsen) and seeking a simple relation
between $S$ and the $A_i$ that has the appropriate symmetry in the $A_i$.
In retrospect this amounts to the quadratic relation on the six
Plücker
coordinates for a $2$-dimensional subspace of ${\bf R}^4$.
To prove the formula for $S'$ we can then argue as follows.
Let $Q_i$ be the vertex of the inner pentagon opposite $P_i$.
Then $S - \sum_{i\,\bmod\,5} A_i$ almost gives the area $S'$ of the
inner pentagon except we must add the areas of the five triangles
$P_{i-2} Q_i P_{i+2}$ which were subtracted twice.  We evaluate
the area of this triangle by using the following observation.
Let $WXYZ$ be a convex quadrilateral whose diagonals intersect at $O$.
Then
$$
{\rm Area}(XOY) = \frac{{\rm Area}(WXY) {\rm Area}(XYZ)} {{\rm Area}(WXYZ)}.
$$
(Proof: use the area formula $\frac12 ab\sin C$ for the areas of
$WOX$, $XOY$, $YOZ$, $ZOW$.)  In our setting $WXYZ$ is
$P_{i+1}P_{i+2}P_{i-2}P_{i-1}$ and its area is $S-A_i$.
The formula above for $S'$ is less pleasant to work with
than it might look because each denominator $S-A_i$ contains
a square root.  It might be better to use as coordinates the quadrilateral areas
$$
B_i := S-A_i = \frac12(P_{i+1} - P_{i-2}) \times (P_{i-1} - P_{i+2})
$$
instead of $A_i$.  Since the relation
between $S$ and the $A_i$ is homogeneous quadratic, the relation between
$S$ and the $B_i$ is also homogeneous quadratic, and indeed it turns out
to be given by the same polynomial
$$
S^2 - S \sum_{i\,\bmod\,5} B_i + \sum_{i\,\bmod\,5} B_i B_{i+1} = 0,
$$
though this time $S$ is the smaller root.
So now we are to prove that the ratio between
$$
S' = S - \sum_{i\,\bmod\,5} (S-B_i)
 + \sum_{i\,\bmod\,5} \frac{(S-B_{i-2}) (S-B_{i+2})}{B_i}.
$$
and $S$ is maximal when all the $B_i$ are equal.
