~~Not an answer, just a long comment.~~ This comment slowly evolved into a partial *numerical* solution: *the regular pentagon is the local maximum*.

I've used Maple to produce the formula for $\frac{S'}{S}$ and the result is a rational function on $\mathbb{R}^{10}$ with both numerator and denominator of degree 12. The numerator has 11495 monomial terms while the denominator is factored into product of six degree two polynomials. This means that the symmetry of the problem is somehow lost during the computation and so one must be a bit careful when using such programs.

Nevertheless, restricting the pentagon to have the first side on the $x$-axes I get numerator that factorizes as product of $x_2$ and a polynomial with 182 monomials. While analytical solution is still most probably out of the question, it may be possible to use this for numerical experiments.

For example, working on $\mathbb{R}^{10}$, I was able to verify that the regular pentagon is a stationary point. To my surprise, Maple computed the eigenvalues of the Hessian at this point analytically. Four of the eigenvalues are negative, but unfortunately there are six eigenvalues that are numerically zero ($\approx 10^{-9}$).

I think it could be possible to "prove" this way that the regular pentagon is a local maximum. I'll try to look at it again tomorrow.

Edit: Let me summarize the comments so far.

As David Speyer mentioned, all pentagons are affinely congruent to those whose vertices lies on a conic. If one uses the rational parametrization of the circle $t \to \left(\frac{t}{1+t^2}, \frac{1-t^2}{1+t^2}\right)$, then the degree of the resulting polynomials in the Jacobian rise to more than 30. Nevertheless, one can factor out terms like $t$ or $t^2 + 1$. Of course, we can still fix one vertex and surprisingly, it matters (at least in Maple) which one. Fixing $(0,1)$ as a vertex leads to better results with degrees $(29,29,29,29)$ and number of monomials $(2348,2754,2754,2348)$. This is still unsolvable (at least in Maple on my computer). But weep not for we can at least compute the Hessian at the regular pentagon and get it's eigenvalues (in this case only numerically, analytic computation timed out): $-0.174, -0.075, -0.001, -0.003$.

*We can conclude (if we believe Maple) that the regular pentagon is the local maximum among "circle pentagons".*

Dag Oskar Madsen reminded me that I didn't use all the freedom we have in the problem. By fixing $(0,0), (1,0)$ and $(0,1)$ I obtained rational function in four variables with numerator of degree 8 with 53 terms and denominator of degree 10 factored into a product of 6 terms. The numerators of the Jacobian have the following structure: "Degrees:", [16, 16, 15, 15], "Terms:", [637, 633, 424, 510] from which we see that this resulted in smallest system so far. On the other hand, any symmetry in the equations was most probably destroyed and thus it may be harder to solve it.

Fixing vertices $1,2,3$ of the regular pentagon with edge $12$ on the $x$-axis I verified that the eigenvalues of the Hessian are indeed negative, namely $-0.027, -0.012, -0.148, -0.136$. *This establishes numerically that the regular pentagon is a local maximum in the set of all pentagons.*

I've tried to play with reformulation by Noam D. Elkies and treat it like a constrained maximization problem. That is
$$
S = \frac{1}{2}\left(a + \sqrt{a^2 - 4b}\right) \\
S' = S - a + \sum_{i \text{ mod } 5} \frac{A_{i-2}A_{i+2}}{S-A_i},\\
\text{where}\\
a = \sum_{i \text{ mod } 5} A_i, \quad b = \sum_{i \text{ mod } 5} A_iA_{i+1}
$$
and $A_i$ is the area of triangle formed by three consecutive vertices $P_{i-1}P_iP_{i+1}$. The problem is to maximize $S'$ relative to $S=1$.

The system for Lagrange multipliers reduces to 5 rational equations with numerators of degree 12 with 1544 terms
$$
\frac{\partial L}{\partial a_i} = f_i + \left(\lambda + \sum_{j \text{ mod } 5} \frac{A_{j-2}A_{j+2}}{(1-A_j)^2} \right)g_i =0,
$$
where
$$
g_i = \frac{1}{2}\left( 1 + \frac{A_i - 4A_{i-1} - 4A_{i+1}}{2-a}\right)\\
f_i = g_i -1 + \sum_{j \text{ mod } 5} \frac{ (A_{j-2}\delta_{i,j+2} + A_{j+2}\delta_{i,j-2})(1-A_j) - A_{j-2}A_{j+2}(g_i - \delta_{i,j}) }{(1-A_j)^2};
$$
plus the constraint
$$
a - b = 1,
$$
which is quadratic with eleven terms.

I encourage everybody to try this out in their CAS of choice. I got curious about the current state of the art in Groebner basis and asked another question.

2more comments