Aaron and fedja have pointed out that the problem is equivalent to finding the convex region in the plane with area $1/2$ with the highest probability that a random line does not intersect it.

The optimal convex region $\Delta$ has boundary a union of eight segments, each satisfying a differential equation from a certain one-parameter family, that hence are smooth.

Pick a corner of the square, choose coordinates so that that corner is point $(0,0)$, and consider the segment of the boundary $C$ of $\Delta$ whose tangent lines touch the two sides of the square adjacent to that point.

If we write this segment of $C$ as the graph of a decreasing function $y(x)$, then the tangent line at the point $(x,y)$ connects the points $\left(x- \frac{y}{\dot{y}},0\right)$ and $\left(0, y+ \dot{y}x \right)$ on these two sides, where $\dot{y}$ is the derivative with respect to $x$. So if we plot the region in the $a,b$ plane consisting of those $(a,b)$ such that the line connecting $(a,0)$ and $(b,0)$ does not intersect $\Delta$, the boundary of that region is the parameteric curve $\left(x- \frac{y}{\dot{y}}, y- \dot{y}x \right)$ and thus the area of the region is $$ - \int \left(x- \frac{y}{\dot{y}} \right) \frac{d}{dx} \left(y - \dot{y}x \right) dx$$

$$= - \int \left(x- \frac{y}{\dot{y}} \right) \left(\frac{dy}{x} - \ddot{y}x - \dot{y} \right) dx$$

$$= \int x \ddot{y} \left(x- \frac{y}{\dot{y}} \right) dx$$

the negative sign being because $\left(y- \dot{y}x \right)$ is a decreasing function of $x$ by convexity.

So we are optimizing $$ \int x \ddot{y} \left(x- \frac{y}{\dot{y}} \right) dx$$ subject to an upper bound on $ \int y dx$ which by Lagrange multipliers is equivalent to optimizing

$$ \int x \ddot{y} \left(x- \frac{y}{\dot{y}} \right) dx - \lambda \int y dx$$

for some $\lambda>0$.

By calculus of variations, if we set $F(y, \dot{y}, \ddot{y}) = x \ddot{y} \left(x- \frac{y}{\dot{y}} \right) - \lambda y$, then the optimal value of $y$ satisfies $$\frac{dF}{dy} - \frac{d}{dx} \left( \frac{dF}{d \dot{y}} - \frac{d}{dx} \left(\frac{dF}{d \ddot{y}} \right) \right) =0$$

We can evaluate
$$\frac{dF}{d \ddot{y}} = x^2 - \frac{xy}{\dot{y}}$$ $$\frac{d}{dx} \left(\frac{dF}{d\ddot{y}} \right)= 2x-\frac{y}{\dot{y}} - x + \frac{xy \ddot{y}}{\left(\dot{y}\right)^2}$$
$$ \frac{dF}{ d\dot{y}} =\frac{xy \ddot{y}}{\left(\dot{y}\right)^2}$$
$$ \frac{dF}{d \dot{y}} - \frac{d}{dx} \left(\frac{dF}{d \ddot{y} }\right) = \frac{y}{\dot{y}} -x $$
$$ \frac{d}{dx} \left( \frac{dF}{d \dot{y}} - \frac{d}{dx} \left(\frac{dF}{d \frac{dy^2}{dx^2}}\right) \right) = 1 - \frac{y \ddot{y}}{\left( \dot{y}\right)^2} -1$$

$$\frac{dF}{dy} = - \lambda - x \frac{\ddot{y}}{\dot{y}}$$

so the differential equation is

$$ - \lambda - x \frac{\ddot{y}}{\dot{y}} + \frac{y \ddot{y}}{\left( \dot{y}\right)^2} =0$$

or

$$ \lambda \left( \dot{y}\right)^2 +( x \dot{y} -y )\ddot{y} =0$$

If we let $t= \dot{y} \frac{x}{y}$ be the dimensionless derivative, then $\dot{y} = t\frac{y}{dx}, \ddot{y} = \frac{d}{dx}\left( t\frac{y}{x}\right)= \dot{t}\frac{y}{x} + t^2 \frac{y}{x^2} - t \frac{y}{x^2} =\frac{y}{x^2} \left( \frac{dt}{d\log x} +t^2-t\right)$

so we can write the equation (ignoring factors of $y$ or $x$) as

$$ \lambda t^2 + (t-1) \left( \frac{dt}{d\log x} + t^2-t \right) =0$$

$$\frac{dt}{d \log x} = - \lambda \frac{t^2}{t-1} + t -t^2 $$

so either we have $t$ a constant solution of $(t^2-t)(t-1) + \lambda t^2 =0$ with $y$ a constant times $x^t$ or we can express $\log x$ and $\log y$ as integrals of rational functions of $t$.

$$\log x = \int \frac{1}{ - \lambda \frac{t^2}{t-1} + t -t^2} dt$$

$$\log y = \int \frac{t}{ - \lambda \frac{t^2}{t-1} + t -t^2} dt$$

Matt F. in the comments did the integrals and found that the formulas, while explicit, are quite nasty. Perhaps this can be fixed by changing the parameter, but this seems unlikely.

It should be possible to do similar calculations for the other kind of segment, but the next step would be to calculate the different ways these segments can be stitched together, which amounts to solving an equation involving eight of these explicit solutions. That seems difficult unless the solutions are really nice - although I'm sure it can be done with the aid of a suitable computer algebra system.

opposingfor your bound to hold? But the problem statement does not guarantee that condition. $\endgroup$ – David G. Stork Jul 17 '17 at 22:34