Unless I misunderstand your question, you are seeking to pack one irregular polygon into another. This has been extensively studied under the name *the polygon containment problem*. It is usually addressed in the context of packing several given polygons into a container. Here is one paper that could lead you to that literature.

> Milenkovic, Victor J. "Rotational polygon overlap minimization and compaction." *Computational Geometry* 10, no. 4 (1998): 305-318.
[PDF download](http://www.cs.miami.edu/home/vjm/Papers/rpcme.pdf).
<hr />
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 
<img src="https://i.sstatic.net/fCtEy.png" width="400" />
<br />
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 
<sup>
Fig.7.
</sup>
<hr />

Use Google Scholar to look through the ~50 later papers that cite this one.


  [1]: https://i.sstatic.net/fCtEy.png