Say that one polygon $P$ is inscribed in another one $Q$, if $P$ is contained entirely in (the interior and boundary of) $Q$ and every vertex of $P$ lies on an edge of $Q$. It's clear that a regular $m$-gon can be inscribed in a regular $n$-gon if $m\mid n$ or $m=2n$. Well-known results (e.g. the Square Peg problem) imply that it is always possible when $m$ is 3 or 4, and it's not hard to use the Intermediate Value Theorem to see that it can happen whenever $m\mid 2n$. It's also not hard to see that a regular pentagon cannot be inscribed in an equilateral triangle, square, or regular hexagon.

Is it ever possible to inscribe a regular $m$-gon in a regular $n$-gon when $m>4$ and $n$ are relatively prime? If not, are there even any cases where it can happen that are not on the above list? Note that this question is not original; it was asked for example as P220 of The Playground in Math Horizons, vol. 16, no. 1, 2008 Sep. But no answer was ever published in Math Horizons and I have not been able to find any other reference.

  • $\begingroup$ Cross-posted from math.stackexchange.com/questions/4492467, where it got no feedback. I hope this is not inappropriate for MO (my first post here). Thanks for any guidance. $\endgroup$ Commented Mar 23 at 1:25
  • $\begingroup$ Re, waiting a suitable time, then cross-posting with a link, is the right etiquette. Neat question! TeX note: $m|n$ m|n spaces poorly, whereas $m\mid n$ m\mid n spaces as more usually expected. I edited accordingly. $\endgroup$
    – LSpice
    Commented Mar 23 at 1:38
  • $\begingroup$ If $m>4$ is replaced yo "$m$ is large enough" this seems to follow from general mumbo-jumbo, I may write details if you care $\endgroup$ Commented Mar 23 at 17:09
  • $\begingroup$ Sure, I'd be happy for an argument that only works for large $m,n$, if that's what you mean. $\endgroup$ Commented Mar 23 at 19:34

1 Answer 1


Here is a proof for large enough coprime $m$ and $n$.

I use some basic properties of cyclotomic polynomials, be free to ask for details if needed. In particular, I use that the sum of roots of $\Phi_k$ equals $\mu(k)$ (Möbius function).

The main result which I use may be phrased as follows: if a sum of $O(1)$ roots of unity (not necessarily distinct) equals 0, this sum is a disjoint collection of zero subsums in each of which your roots of unity are some vertices of $O(1)$-gon.

This certainly must be known, but I do not know the reference and reproduce below a (short) proof, in slightly different form.

Let $N$ be a positive integer, Denote $\xi=e^{2\pi i/N}$. For an element $x\in \mathbb{Q}[\xi]$, i.e., $x=f(\xi)$ with $f(t)\in \mathbb{Q}[t]$, we denote the normalized trace $$T(x)=\frac1{\varphi(N)}\sum_{(k,N)=1}f(\xi^k),$$ where the summation is taken over all $\varphi(N)$ residues $k$ modulo $N$ which are coprime to $N$. Note that although the polynomial $f(t)$ for given $x$ is not uniquely defined, it is defined uniquely modulo the cyclotomic polynomial $\Phi_N(t)$, which is a minimal polynomial of the algebraic number $\xi$. Thus, $f(\xi^k)$ are well defined for all $k$ coprime to $N$ (the numbers $\xi^k$ are roots of $\Phi_N$, in other words, algebraic conjugates of $\xi$), and $T(x)$ is well-defined. Further we also use not polynomials, but Laurent polynomials $f(\xi)$, which does not abuse generality, since we may always replace negative powers of $\xi$ by non-negative powers of $\xi$ using $\xi^{-1}=\xi^{N-1}$.

If $f(t)=t^r$, then the numbers $f(\xi^k)$, where $k$ is coprime to $N$, are the roots of $\Phi_{N/(r,N)}$, each counted $\varphi(N)/\varphi(N/(r,N))$ times. Therefore $T(\xi^r)=\frac{\mu(N/(r,N))}{\varphi(N/(r,N))}$, for $r=N$ this gives $T(1)=1$. The idea is that the trace $T(\xi^r)$ is small when $N/(r,N)$ is large.

Consider the expression of the type $h(\xi)=\sum_{j\in A} c_j\xi^j$, where $A$ is a finite set of integers, $c_j\ne 0$ for $j\in A$ are integers.

Lemma. $h(\xi)=0$ if and only if $T(\xi^{-j}h(\xi))=0$ for all $j\in A$.

Proof. "Only if" part is trivial. For proving "if" part, by linearity of $T$ we see that these relations yield $T((\sum_j c_j \xi^{-j})h(\xi))=0$, but for Laurent polynomial $g(t):=(\sum_j c_j t^{-j})h(t)$ all values on the unit circle are non-negative real numbers, and $T(g(\xi))=0$ yields $|h(\xi)|^2=g(\xi)=0$.

Assume now that $h(\xi)=0$ in the above notations for $h(\xi)=\sum_{j\in A} c_j\xi^j$ and $\sum |c_j|=O(1)$ while $N$ may be arbitrarily large. Then we may suppose (by passing a subsequence) that $A=\{j_1,\ldots,j_R\}$ for fixed $R$, $c_{j_1},\ldots,c_{j_R}$ are also fixed. Also, we may suppose that, for every $1\leqslant a,b\leqslant R$, the normalized trace $T(\xi^{j_a-j_b})$ is either fixed or goes to 0 when $m,n$ become large.

Join two elements $j_1,j_2$ of $A$ by an edge if $T(\xi^{j_a-j_b})$ takes a fixed non-zero value. Let, say, $\{j_1,\ldots,j_r\}$ (where $r\leqslant R$) form a connected component. In this component, the ratio $\xi^{j_a-j_b}$ of every two numbers $\xi^{j_1},\ldots,\xi^{j_r}$ is a root of unity of bounded degree. I claim that $h_1(\xi)=\sum_{a=1}^r c_{j_a}\xi^{j_a}$ is zero for large $n,m$. For proving this, note that for $a\leqslant r$ the numbers $T(\xi^{-j_a}h_1(\xi))$ and $0=T(\xi^{-j_a}h(\xi))$ differ by $o(1)$. Therefore the former is 0 for large $N$, as it may take only finitely many different values. It remains to apply Lemma.

Now back to your $m$-gon and $n$-gon. Denote $N=mn$, $\xi=e^{2\pi i/N}$, $\omega=e^{2\pi i/m}=\xi^n$, $\theta=e^{2\pi i/n}=\xi^m$. Every side of the $n$-gon (which is assumed to be formed by powers of $\theta$) may contain at most 2 vertices of the $m$-gon (which is assumed to be formed by the numbers of the form $A\omega^j+B$ for some complex constants $A\ne 0$ and $B$). Let $A\omega^j+B$ belongs to the line between $\theta^{n_j}$ and $\theta^{n_j+1}$, $0\leqslant n_j<n$.

We may find a set $I$ of five distinct indices $j$ between 0 and 8 for which and all $n_j$, $j\in I$, are distinct. The condition that $A\omega^j+B$ belongs to the line between $\theta^{n_j}$ and $\theta^{n_j+1}$ implies that $p_j:=A\omega^j\theta^{-n_j}+B\theta^{-n_j}$ belongs to a fixed line between 1 and $\theta$ which has equation $\theta \bar{z}+z-(1+\theta)=0$ (it is not important which exact equation, though). Substituting $p_j$ to this equation, we get a linear combination of five numbers $\omega^j\theta^{-n_j}, \theta^{-n_j}, \omega^{-j}\theta^{n_j}, \theta^{n_j},1$ with fixed (and not all 0) coefficients. By fixed, I mean not dependent on $j$. Therefore, the determinant of the $5\times 5$ matrix formed by the above vectors of length 5 equals 0. This determinant $D$ is a linear combination of at most 120 roots of unity of degree $N$ with coefficients $\pm 1$. Note that the number of the form $\omega^J\theta^M$ with $0<|J|\leqslant 8$ and $M$ is arbitrary is not a root of unity of bounded degree if $m\to \infty$. Therefore, the above argument yields that if we formally expand the determinant and recollect the terms in the form $D=\sum_{J=-8}^8 \omega^J f_J(\theta)$, then each specific $f_j(\theta)$ must vanish (for large $m,n$). But, if $I=\{j_1<j_2<j_3<j_4<j_5\}$, then for $J=j_5-j_1$ the guy $f_J(\theta)$ is $\theta^{j_1-j_5}$ times a $3\times 3$ determinant with rows $(\theta^{n_j},\theta^{-n_j},1)$ for $j=2,3,4$. Its value is non-zero, since if you multiply the row $(\theta^{n_j},\theta^{-n_j},1)$ by $\theta^{n_j}$ for all $j=2,3,4$, you get a Vandermonde determinant for three distinct numbers $\theta^{n_2},\theta^{n_3},\theta^{n_4}$.

  • $\begingroup$ Thank you for this insight! The core seems to be the reduction that if m-gon is inscribed in n-gon, there is a vanishing sum of roots of unity of form $\sum \omega^j L_j$ where $|j| <9$ and each $L_j$ is of the form $\sum \pm\theta^k$ (using your $\omega$ and $\theta$). At that point, can't we just finish by invoking the de Bruijn theorem that says that such a vanishing sum must be integer linear combination of $\beta \sum_{i=0}^{p-1} \alpha^i$ for $\alpha$ prim $p$th root of unity and $\beta$ an arbitrary root of unity? $\endgroup$ Commented Mar 24 at 22:21
  • $\begingroup$ Because then we actually can't have $p\mid m$, because if so each term must end up in a different $\omega^j$ portion of the original vanishing sum, but there are not enough such terms (when $m \geq 37$). So therefore $p\mid n$, and the whole linear combination ends up in the same $L_j$. Hence each $L_j$ is a linear combination of these primitive vanishing sums, and so vanishes, as desired. In other words, $m\geq 37$ is large enough for $m$-gon not to inscribe in any coprime $n$-gon. Does that seem correct? $\endgroup$ Commented Mar 24 at 22:26
  • $\begingroup$ Actually, I think 37 in my last comment should have been 35. The $\omega^j$ hit at most the 17 congruence classes mod $m$ from -8 to 8, so any rotation of a primitive relation mod $p\mid m$ would contain an exponent not equivalent to one of these as long as there are at least $2\cdot 17 + 1$ congruence classes mod $m$. $\endgroup$ Commented Mar 24 at 23:15
  • $\begingroup$ Darn, that's not correct, as there can be cancellation from different primes $p\mid m$. E.g., if $6\mid m$ then $\xi^{m/6} - 1 + \xi^{-m/6} = 0$, and those three congruence classes are contained in $[-8, 8]$ for $m$ up to 48. So now I am not sure if there is a way to finish off with the de Bruijn theorem. $\endgroup$ Commented Mar 25 at 1:39
  • $\begingroup$ I guess we at least get the much weaker outcome from just de Bruijn that if $m$ has no prime factor < 17, and $n$ is relatively prime to $m$, then the inscribing cannot occur, because there are at most 16 non-empty components $L_j$. $\endgroup$ Commented Mar 25 at 3:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.