Here is a partly baked idea: Hang a sheet--a light cloth--from the segment $(0,0,0)-(1,0,0)$ along the bottom of your square, and from the segment $(0,1,0)-(1,1,1)$ slanting above the top of your square, and let it sag under gravity below the $xy$-plane, pinned to these two segments. There are very nice cloth simulation algorithms implemented, e.g., in Blender below. It seems you could approximate the shape of the sheet by hanging a catenary between the points $(x,0,0)$ and $(x,1,x)$. By adjusting the lengths of the catenaries as a function of $x$, it seems you should be able to make the surface concave throughout.
![Cloth simulation][2]