Unless I misunderstand your question, you are seeking to pack one irregular polygon into another. This has been extensively studied under the name the polygon containment problem. It is usually addressed in the context of packing several given polygons into a container. Here is one paper that could lead you to that literature.
Milenkovic, Victor J. "Rotational polygon overlap minimization and compaction." Computational Geometry 10, no. 4 (1998): 305-318. PDF download.
Fig.7.
Use Google Scholar to look through the ~50 later papers that cite this one.