Unless I misunderstand your question, you are seeking to pack one irregular polygon into another. This has been extensively studied under the name the polygon containment problem:
Chazelle, Bernard. The polygon containment problem. Carnegie-Mellon University, Department of Computer Science, 1981. Proves that the problem can be solved in polynomial-time, about $O(n^7)$ in the general case, for polygons of $n$ vertices.
The problem is usually addressed in the context of packing several given polygons into a container. Here is one paper that could lead you to that literature (Google Scholar lists ~50 later papers that cite this one):
Milenkovic, Victor J. "Rotational polygon overlap minimization and compaction." Computational Geometry 10, no. 4 (1998): 305-318. PDF download.
Fig.7.