Inscribing one regular polygon in another

Say that one polygon $$P$$ is inscribed in another one $$Q$$, if $$P$$ is contained entirely in (the interior and boundary of) $$Q$$ and every vertex of $$P$$ lies on an edge of $$Q$$. It's clear that a regular $$m$$-gon can be inscribed in a regular $$n$$-gon if $$m\mid n$$ or $$m=2n$$. Well-known results (e.g. the Square Peg problem) imply that it is always possible when $$m$$ is 3 or 4, and it's not hard to use the Intermediate Value Theorem to see that it can happen whenever $$m\mid 2n$$. It's also not hard to see that a regular pentagon cannot be inscribed in an equilateral triangle, square, or regular hexagon.

Is it ever possible to inscribe a regular $$m$$-gon in a regular $$n$$-gon when $$m>4$$ and $$n$$ are relatively prime? If not, are there even any cases where it can happen that are not on the above list? Note that this question is not original; it was asked for example as P220 of The Playground in Math Horizons, vol. 16, no. 1, 2008 Sep. But no answer was ever published in Math Horizons and I have not been able to find any other reference.

• Cross-posted from math.stackexchange.com/questions/4492467, where it got no feedback. I hope this is not inappropriate for MO (my first post here). Thanks for any guidance. Commented Mar 23 at 1:25
• Re, waiting a suitable time, then cross-posting with a link, is the right etiquette. Neat question! TeX note: $m|n$ m|n spaces poorly, whereas $m\mid n$ m\mid n spaces as more usually expected. I edited accordingly. Commented Mar 23 at 1:38
• If $m>4$ is replaced yo "$m$ is large enough" this seems to follow from general mumbo-jumbo, I may write details if you care Commented Mar 23 at 17:09
• Sure, I'd be happy for an argument that only works for large $m,n$, if that's what you mean. Commented Mar 23 at 19:34

Here is a proof for large enough coprime $$m$$ and $$n$$.

I use some basic properties of cyclotomic polynomials, be free to ask for details if needed. In particular, I use that the sum of roots of $$\Phi_k$$ equals $$\mu(k)$$ (Möbius function).

The main result which I use may be phrased as follows: if a sum of $$O(1)$$ roots of unity (not necessarily distinct) equals 0, this sum is a disjoint collection of zero subsums in each of which your roots of unity are some vertices of $$O(1)$$-gon.

This certainly must be known, but I do not know the reference and reproduce below a (short) proof, in slightly different form.

Let $$N$$ be a positive integer, Denote $$\xi=e^{2\pi i/N}$$. For an element $$x\in \mathbb{Q}[\xi]$$, i.e., $$x=f(\xi)$$ with $$f(t)\in \mathbb{Q}[t]$$, we denote the normalized trace $$T(x)=\frac1{\varphi(N)}\sum_{(k,N)=1}f(\xi^k),$$ where the summation is taken over all $$\varphi(N)$$ residues $$k$$ modulo $$N$$ which are coprime to $$N$$. Note that although the polynomial $$f(t)$$ for given $$x$$ is not uniquely defined, it is defined uniquely modulo the cyclotomic polynomial $$\Phi_N(t)$$, which is a minimal polynomial of the algebraic number $$\xi$$. Thus, $$f(\xi^k)$$ are well defined for all $$k$$ coprime to $$N$$ (the numbers $$\xi^k$$ are roots of $$\Phi_N$$, in other words, algebraic conjugates of $$\xi$$), and $$T(x)$$ is well-defined. Further we also use not polynomials, but Laurent polynomials $$f(\xi)$$, which does not abuse generality, since we may always replace negative powers of $$\xi$$ by non-negative powers of $$\xi$$ using $$\xi^{-1}=\xi^{N-1}$$.

If $$f(t)=t^r$$, then the numbers $$f(\xi^k)$$, where $$k$$ is coprime to $$N$$, are the roots of $$\Phi_{N/(r,N)}$$, each counted $$\varphi(N)/\varphi(N/(r,N))$$ times. Therefore $$T(\xi^r)=\frac{\mu(N/(r,N))}{\varphi(N/(r,N))}$$, for $$r=N$$ this gives $$T(1)=1$$. The idea is that the trace $$T(\xi^r)$$ is small when $$N/(r,N)$$ is large.

Consider the expression of the type $$h(\xi)=\sum_{j\in A} c_j\xi^j$$, where $$A$$ is a finite set of integers, $$c_j\ne 0$$ for $$j\in A$$ are integers.

Lemma. $$h(\xi)=0$$ if and only if $$T(\xi^{-j}h(\xi))=0$$ for all $$j\in A$$.

Proof. "Only if" part is trivial. For proving "if" part, by linearity of $$T$$ we see that these relations yield $$T((\sum_j c_j \xi^{-j})h(\xi))=0$$, but for Laurent polynomial $$g(t):=(\sum_j c_j t^{-j})h(t)$$ all values on the unit circle are non-negative real numbers, and $$T(g(\xi))=0$$ yields $$|h(\xi)|^2=g(\xi)=0$$.

Assume now that $$h(\xi)=0$$ in the above notations for $$h(\xi)=\sum_{j\in A} c_j\xi^j$$ and $$\sum |c_j|=O(1)$$ while $$N$$ may be arbitrarily large. Then we may suppose (by passing a subsequence) that $$A=\{j_1,\ldots,j_R\}$$ for fixed $$R$$, $$c_{j_1},\ldots,c_{j_R}$$ are also fixed. Also, we may suppose that, for every $$1\leqslant a,b\leqslant R$$, the normalized trace $$T(\xi^{j_a-j_b})$$ is either fixed or goes to 0 when $$m,n$$ become large.

Join two elements $$j_1,j_2$$ of $$A$$ by an edge if $$T(\xi^{j_a-j_b})$$ takes a fixed non-zero value. Let, say, $$\{j_1,\ldots,j_r\}$$ (where $$r\leqslant R$$) form a connected component. In this component, the ratio $$\xi^{j_a-j_b}$$ of every two numbers $$\xi^{j_1},\ldots,\xi^{j_r}$$ is a root of unity of bounded degree. I claim that $$h_1(\xi)=\sum_{a=1}^r c_{j_a}\xi^{j_a}$$ is zero for large $$n,m$$. For proving this, note that for $$a\leqslant r$$ the numbers $$T(\xi^{-j_a}h_1(\xi))$$ and $$0=T(\xi^{-j_a}h(\xi))$$ differ by $$o(1)$$. Therefore the former is 0 for large $$N$$, as it may take only finitely many different values. It remains to apply Lemma.

Now back to your $$m$$-gon and $$n$$-gon. Denote $$N=mn$$, $$\xi=e^{2\pi i/N}$$, $$\omega=e^{2\pi i/m}=\xi^n$$, $$\theta=e^{2\pi i/n}=\xi^m$$. Every side of the $$n$$-gon (which is assumed to be formed by powers of $$\theta$$) may contain at most 2 vertices of the $$m$$-gon (which is assumed to be formed by the numbers of the form $$A\omega^j+B$$ for some complex constants $$A\ne 0$$ and $$B$$). Let $$A\omega^j+B$$ belongs to the line between $$\theta^{n_j}$$ and $$\theta^{n_j+1}$$, $$0\leqslant n_j.

We may find a set $$I$$ of five distinct indices $$j$$ between 0 and 8 for which and all $$n_j$$, $$j\in I$$, are distinct. The condition that $$A\omega^j+B$$ belongs to the line between $$\theta^{n_j}$$ and $$\theta^{n_j+1}$$ implies that $$p_j:=A\omega^j\theta^{-n_j}+B\theta^{-n_j}$$ belongs to a fixed line between 1 and $$\theta$$ which has equation $$\theta \bar{z}+z-(1+\theta)=0$$ (it is not important which exact equation, though). Substituting $$p_j$$ to this equation, we get a linear combination of five numbers $$\omega^j\theta^{-n_j}, \theta^{-n_j}, \omega^{-j}\theta^{n_j}, \theta^{n_j},1$$ with fixed (and not all 0) coefficients. By fixed, I mean not dependent on $$j$$. Therefore, the determinant of the $$5\times 5$$ matrix formed by the above vectors of length 5 equals 0. This determinant $$D$$ is a linear combination of at most 120 roots of unity of degree $$N$$ with coefficients $$\pm 1$$. Note that the number of the form $$\omega^J\theta^M$$ with $$0<|J|\leqslant 8$$ and $$M$$ is arbitrary is not a root of unity of bounded degree if $$m\to \infty$$. Therefore, the above argument yields that if we formally expand the determinant and recollect the terms in the form $$D=\sum_{J=-8}^8 \omega^J f_J(\theta)$$, then each specific $$f_j(\theta)$$ must vanish (for large $$m,n$$). But, if $$I=\{j_1, then for $$J=j_5-j_1$$ the guy $$f_J(\theta)$$ is $$\theta^{j_1-j_5}$$ times a $$3\times 3$$ determinant with rows $$(\theta^{n_j},\theta^{-n_j},1)$$ for $$j=2,3,4$$. Its value is non-zero, since if you multiply the row $$(\theta^{n_j},\theta^{-n_j},1)$$ by $$\theta^{n_j}$$ for all $$j=2,3,4$$, you get a Vandermonde determinant for three distinct numbers $$\theta^{n_2},\theta^{n_3},\theta^{n_4}$$.

• Thank you for this insight! The core seems to be the reduction that if m-gon is inscribed in n-gon, there is a vanishing sum of roots of unity of form $\sum \omega^j L_j$ where $|j| <9$ and each $L_j$ is of the form $\sum \pm\theta^k$ (using your $\omega$ and $\theta$). At that point, can't we just finish by invoking the de Bruijn theorem that says that such a vanishing sum must be integer linear combination of $\beta \sum_{i=0}^{p-1} \alpha^i$ for $\alpha$ prim $p$th root of unity and $\beta$ an arbitrary root of unity? Commented Mar 24 at 22:21
• Because then we actually can't have $p\mid m$, because if so each term must end up in a different $\omega^j$ portion of the original vanishing sum, but there are not enough such terms (when $m \geq 37$). So therefore $p\mid n$, and the whole linear combination ends up in the same $L_j$. Hence each $L_j$ is a linear combination of these primitive vanishing sums, and so vanishes, as desired. In other words, $m\geq 37$ is large enough for $m$-gon not to inscribe in any coprime $n$-gon. Does that seem correct? Commented Mar 24 at 22:26
• Actually, I think 37 in my last comment should have been 35. The $\omega^j$ hit at most the 17 congruence classes mod $m$ from -8 to 8, so any rotation of a primitive relation mod $p\mid m$ would contain an exponent not equivalent to one of these as long as there are at least $2\cdot 17 + 1$ congruence classes mod $m$. Commented Mar 24 at 23:15
• Darn, that's not correct, as there can be cancellation from different primes $p\mid m$. E.g., if $6\mid m$ then $\xi^{m/6} - 1 + \xi^{-m/6} = 0$, and those three congruence classes are contained in $[-8, 8]$ for $m$ up to 48. So now I am not sure if there is a way to finish off with the de Bruijn theorem. Commented Mar 25 at 1:39
• I guess we at least get the much weaker outcome from just de Bruijn that if $m$ has no prime factor < 17, and $n$ is relatively prime to $m$, then the inscribing cannot occur, because there are at most 16 non-empty components $L_j$. Commented Mar 25 at 3:42