Indeed it seems that the circle is not optimal although the improvements I've found only raise the probability slightly and are visually hard to distinguish from a circle. Here is the circle in red and a region $\Delta_h$ bounded by $4$ hyperbolas. Recall that the circle has $p(\Delta_0)=0.34470989\dots.$ It turns out that $p(\Delta_h)=0.34494\dots.$

It was suggested that the optimum might be a shape bounded by $4$ hyperbolas near the corners and $4$ parabolas near the middles of the sides. The best I could do with that is $\Delta_{hp}$ shown here with $p(\Delta_{hp})=.34544686\dots.$

The parabola at the bottom is roughly $y=1.221614(x-\frac12)^2+.1045232.$ The transition from the parabola to the hyperbola to its left occurs between $(0.26957, 0.16938)$ and $(0.267287, .170719)$ (the respective endpoints of the polygonal approximations) with the middle of the hyperbola at $(0.213614,0.213614).$

Some disclaimers: The later digits may not be exact, I tried to truncate before a small calculated digit. These are actually polygons obtained by sampling (at equally radially spaced points). The area may come out to be less than $\frac12$ in which case the figure is dilated appropriately. This will map parabolas to parabolas but deform hyperbolas. However replacing the deformed sections by a best hyperbola approximation changes the area only slightly. Iterating a few times stabilizes. I did not do this adjusting to $\Delta_h$ so it may be cheated out of a bit.

I don't know if this is really the optimal shape. I could investigate small perturbations of the polygon but have not. I am pretty sure that something similar to a quarter circle in a corner OR a parabola along with most of the top OR the top, with the upper portions of the two sides connected by the arc of a conic could not do as well. I'm not saying it would be hard to rule those out. I just haven't rigorously done so.

Indeed it seems that the circle is not optimal although the improvements I've found only raise the probability slightly and are visually hard to distinguish from a circle. Here is the circle in red and a region $\Delta_h$ bounded by $4$ hyperbolas. Recall that the circle has $p(\Delta_0)=0.34470989\dots.$ It turns out that $p(\Delta_h)=0.34494\dots.$

It was suggested that the optimum might be a shape bounded by $4$ hyperbolas near the corners and $4$ parabolas near the middles of the sides. The best I could do with that is $\Delta_{hp}$ shown here with $p(\Delta_{hp})=.34544686\dots.$

The parabola at the bottom is roughly $y=1.221614(x-\frac12)^2+.1045232.$ The transition from the parabola to the hyperbola to its left occurs between $(0.26957, 0.16938)$ and $(0.267287, .170719)$ (the respective endpoints of the polygonal approximations) with the middle of the hyperbola at $(0.213614,0.213614).$

Some disclaimers: I am not certain this is correct. I expected that the contact point of a tangent line rolling around would transition from hyperbola to parabola exactly when its intersection of the line with the square passes a corner and begins to intersect opposite sides. I believe this would maintain a constant area between the tangent and the square.However I don't see that happening here. Also, The later digits may not be exact, I tried to truncate before a small calculated digit. These are actually polygons obtained by sampling (at equally radially spaced points). The area may come out to be less than $\frac12$ in which case the figure is dilated appropriately. This will map parabolas to parabolas but deform hyperbolas. However replacing the deformed sections by a best hyperbola approximation changes the area only slightly. Iterating a few times stabilizes. I did not do this adjusting to $\Delta_h$ so it may be cheated out of a bit.

I don't know if this is really the optimal shape. I could investigate small perturbations of the polygon but have not. I am pretty sure that something similar to a quarter circle in a corner OR a parabola along with most of the top OR the top, with the upper portions of the two sides connected by the arc of a conic could not do as well. I'm not saying it would be hard to rule those out. I just haven't rigorously done so.