Here are two references. The first, a 1981 paper, provides an
algorithm to solve the problem allowing
translations and rotations, but not scaling:
Chazelle, Bernard. "The polygon containment problem." Carnegie-Mellon University, Department of Computer Science, 1981.
This was followed in 1994 by an algorithm that includes scaling.
The algorithm uses parametric search:
Sharir, Micha, and Sivan Toledo. "Extremal polygon containment problems." Computational Geometry-Theory and Application 4, no. 2 (1994): 99.
All algorithms are polynomial in the number of vertices of the two polygons.
In the latter paper, the time complexity is
$O(k^2 n \lambda_6(kn) \log^3(kn) \log\log(kn))$,
where the containing polygon has $n$ vertices and the contained has $k$ vertices. For $k=n$, this is a bit beyond $O(n^5)$.
Various speed-ups can be achieved if one or the other polygon is known
to be convex, or if one restricts the rigid motions allowed.
Google Scholar shows about 50 papers that subsequently cite Sharir-Toledo.
Packing applications have led some researchers to more practical