If $(x,y)$ is the coordinate of the upper right corner of the rectangle, with the origin at the center of the circles (inner radius $r_1$, outer radius $r_2$), then the area of the rectangle is $A=2x(y-r_1)$. Eliminate $y=\sqrt{r_2^2-x^2}$, differentiate $dA/dx$ and solve $dA/dx=0$ for $x$. For $r_1=50$, $r_2=60$ this gives the maximal area $$A_{\rm max}=\tfrac{1}{4} \sqrt{11900-500 \sqrt{313}} \left(\sqrt{500 \sqrt{313}+16900}-100 \sqrt{2}\right)=262.972$$
More generally, $$A_{\rm max}=\frac{1}{4} \sqrt{-r_1^2+4 r_2^2-\sqrt{r_1^4+8 r_1^2 r_2^2}} \left(-2 \sqrt{2}\, r_1+\sqrt{r_1^2+4 r_2^2+\sqrt{r_1^4+8 r_1^2 r_2^2}}\right).$$
Carlo Beenakker
- 188.1k
- 18
- 448
- 651