Indeed it seems that the circle is not optimal although the improvements I've found only raise the probability slightly and are visually hard to distinguish from a circle. Here is the circle in red and a region $\Delta_h$ bounded by $4$ hyperbolas. Recall that the circle has $p(\Delta_0)=0.34470989\dots.$ It turns out that $p(\Delta_h)=0.34494\dots.$
It was suggested that the optimum might be a shape bounded by $4$ hyperbolas near the corners and $4$ parabolas near the middles of the sides. The best I could do with that is $\Delta_{hp}$ shown here with $p(\Delta_{hp})=.34544686\dots.$
The parabola at the bottom is roughly $y=1.221614(x-\frac12)^2+.1045232.$ The transition from the parabola to the hyperbola to its left occurs between $(0.26957, 0.16938)$ and $(0.267287, .170719)$ (the respective endpoints of the polygonal approximations) with the middle of the hyperbola at $(0.213614,0.213614).$
Some disclaimers: The later digits may not be exact, I tried to truncate before a small calculated digit. These are actually polygons obtained by sampling (at equally radially spaced points). The area may come out to be less than $\frac12$ in which case the figure is dilated appropriately. This will map parabolas to parabolas but deform hyperbolas. However replacing the deformed sections by a best hyperbola approximation changes the area only slightly. Iterating a few times stabilizes. I did not do this adjusting to $\Delta_h$ so it may be cheated out of a bit.
I don't know if this is really the optimal shape. I could investigate small perturbations of the polygon but have not. I am pretty sure that something similar to a quarter circle in a corner OR a parabola along with most of the top OR the top, with the upper portions of the two sides connected by the arc of a conic could not do as well. I'm not saying it would be hard to rule those out. I just haven't rigorously done so.