Is it known which plane figures $Q$ can rotate touching three given circles $A$, $B$, and $C$?
This question was asked by Lazar Lyusternik in 1946, there is only one reference to this paper that solves the problem in one limit case.
Do you know of any other research on this question?
(I learned this question from Sergei Tabachnikov.)
I have looked at Goldberg's paper referenced by J. J. Castro in his excellent answer. It turns out that there is a simpler (and more general way) to generate Goldberg's non-circular solutions, so I thought that I would just mention that.
The idea is to consider the 'rotor' (Goldberg's term) as fixed and as the envelope of a circle in periodic motion. It's a bit easier if you use complex notation, i.e., think of $\mathbb{R}^2$ as $\mathbb{C}$. Consider a circle of radius $r>0$ and center $z_0=1$, parametrized by $z(s) = 1 + r\mathrm{e}^{is}$, and move it by a circle of rigid motions: $R_t(z) = \mathrm{e}^{it}z + a(t)$, where $a$ is periodic of period $2\pi/n$ for some integer $n\ge 3$. Now take the two envelopes of this $1$-parameter family of circles $R_t\bigl(z(s)\bigr)$. Because of the periodicity of $a$, these will also be envelopes of the circles of radius $r$ centered at the other $n$-th roots of unity, and hence these envelopes will have the property that, when they are moved by the inverse of $R_t$, they will remain tangent to the $n$ circles of radius $r$ centered on the $n$-th roots of unity.
A simple computation shows that the two envelopes of this family are given by $$ E_\pm(t) = \mathrm{e}^{it}+a(t) \pm \frac{r\bigl(\mathrm{e}^{it}-ia'(t)\bigr)}{\bigl|\mathrm{e}^{it}-ia'(t)\bigr|}. $$ (Note that this is well-defined as long as $\mathrm{e}^{it}-ia'(t)$ never vanishes, which can be ensured, for example, by choosing $a$ so that $|a'(t)|<1$.)
Setting $a=0$ (or a constant) gives the trivial circle solution. Below, I give an example of $E_-$ (the interior envelope) with $n=3$, $r=1/2$, and $a(t) = \tfrac17\sin(3t)$, or, more precisely, $R_{-s}(E_-)$ as $s$ varies between $0$ and $2\pi$. (Replacing this by $a(t) = \tfrac16\sin(3t)$ yields a nonconvex solution. In general, when $a$ is allowed to be large, the envelopes will have cusps.)
Lyusternik´s problem was proposed also in the book by Yaglom and Boltyanski "Convex Figures". There is another case which was solved by Goldberg, when the circles (3 or more circles) are of the same radii and its centers are in the vertices of a regular polygon. https://onlinelibrary.wiley.com/doi/abs/10.1002/sapm195837169
