If $(x,y)$ is the coordinate of the upper right corner of the rectangle, with the origin at the center of the circles (inner radius $r_1$, outer radius $r_2$), then the area of the rectangle is $A=2x(y-r_1)$. Eliminate $y=\sqrt{r_2^2-x^2}$, differentiate $dA/dx$ and solve $dA/dx=0$ for $x$. For $r_1=50$, $r_2=60$ this gives the maximal area $$A_{\rm max}=75 \sqrt{1854-70 \sqrt{313}}=1860.81$$
More generally, the maximal area is the largest of $A_+$ and $A_-$, with $$A_{\pm}=\pm\frac{1}{4} \sqrt{-r_1^2\pm\sqrt{r_1^4+8 r_1^2 r_2^2}+4 r_2^2} \left(2 \sqrt{2} r_1\mp\sqrt{r_1^2+\sqrt{r_1^4+8 r_1^2 r_2^2}+4 r_2^2}\right).$$
Carlo Beenakker
- 188.1k
- 18
- 448
- 651