I came across a problem concerning about the convergence of products. I wonder if the complex series $a_n=\prod^n_{k=1}(1-e^{k\alpha \pi i})$ converges to zero when $\alpha$ is irrational. Of course, the infinite product doesn't converge to any nonzero number.
It seems that the behavior of $e^{k\alpha \pi i}$ is not quite "predictable". I have no idea how to approach this problem. Thanks in advance!
The product does not tend to the limit zero. For any irrational number $\alpha$ one can show that $$ \limsup_{N\to \infty} \prod_{n=1}^{N} |1- e(n\alpha)| = \infty. \tag{1} $$ (Here I use the usual notation $e(\alpha) =e^{2\pi i\alpha}$, and the product in the question is stated for $\alpha/2$ rather than $\alpha$).
I'll prove a slightly weaker result; namely I'll show that (1) holds for any irrational $\alpha$ for which there are infinitely many rational approximations $a/q$ with $(a,q)=1$ and $$ \alpha=\frac{a}{q} +\beta, \qquad \text{with}\qquad |\beta| \le \frac{1}{100q^2}. \tag{2} $$ The $1/100$ is just chosen for ease of exposition, and a more careful argument can make do with $1/\sqrt{5}$ (any constant $<1/2$ is enough), and then it is well known that every irrational number admits infinitely many approximations with $|\alpha -a/q| \le 1/(\sqrt{5}q^2)$. Condition 2 holds for almost all irrational numbers -- it fails only if the continued fraction expansion only uses numbers below $100$.
Suppose then that $q$ is such that (2) holds, and consider the product in question at $N=q-1$. By the triangle inequality, for $1\le n\le N$, $$ |1-e(n\alpha)| = |(1-e(an/q))+e(an/q)(1-e(\beta n))| \ge |1-e(an/q)| \Big(1-\frac{|1-e(n\beta)|}{|1-e(an/q)|}\Big)= 2\Big|\sin\frac{\pi an}{q}\Big| \Big( 1- \frac{|\sin(\pi n\beta)|}{|\sin(\pi an/q)|}\Big). $$ Now write $\Vert x\Vert= \min_{\ell \in {\Bbb Z}} |x-\ell|$ for the distance from $x$ to the nearest integer. Note also that for $0\le x\le \pi/2$ one has $2x/\pi \le \sin x \le x$. Therefore we get for $1\le n\le N$ $$ |1-e(n\alpha)| \ge |1-e(an/q)| \Big(1 - \frac{\pi n|\beta|}{2\Vert an/q\Vert}\Big) \ge |1-e(an/q)| \exp\Big( - \frac{\pi q|\beta|}{\Vert an/q\Vert}\Big), \tag{3} $$ where in the last inequality we used that $\eta =\pi n|\beta|/(2\Vert an/q\Vert) \le \pi q^2|\beta|/2 \le 1/10$, so that $(1-\eta) \ge \exp(-2\eta)$.
Multiplying (3) for $n$ from $1$ to $N=q-1$, we obtain $$ \prod_{n=1}^{N} |1-e(n\alpha)| \ge \prod_{n=1}^{q-1} |1-e(an/q)| \exp\Big( - \sum_{n=1}^{N} \frac{\pi q |\beta|}{\Vert an/q\Vert}\Big). \tag{4} $$ Now $$ \sum_{n=1}^{q-1} \frac{1}{\Vert an/q\Vert} \le 2\sum_{n\le q/2} \frac{q}{n} =2 q \log q +O(q), $$ and $\prod_{n=1}^{N} |1-e(an/q)| = q$, and so from (4) we conclude that $$ \prod_{n=1}^{q-1} |1-e(n\alpha)| \ge q \exp\Big( - 2\pi q^2 |\beta| \log q +O(q^2|\beta|)\Big) \gg q^{0.9}. $$
This proves the claim. Note that this argument is closely related to the nice attempt of Sangchul Lee.
Added information (October, 2021): From comments on Fedja's recent question Can all partial sums $\sum_{k=1}^n f(ka)$ where $f(x)=\log|2\sin(x/2)|$ be non-negative? I was led to the following paper by D.S. Lubinsky which shows that (see Theorem 1.2 there) $$ \limsup_{N\to \infty} \frac{\log \prod_{n=1}^{N} |1-e(n\alpha)|}{\log N} \ge 1, $$ for all irrational numbers $\alpha$, so that the product in the question gets almost as large as $N$ infinitely often.
I have no idea how to approach the general problem, but here is a quick observation:
A. Let $\alpha = 2\beta$ so that $\beta$ is irrational if and only if $\alpha$ is so. Define $f$ by $f(x) = \log|2\sin\pi x|$. Then
$$ \log \left| \prod_{k=1}^{n} (1 - e^{\pi k i \alpha} ) \right| = \sum_{k=1}^{n} f(k\beta). $$
Now by the Riemann-integrable criterion for equidistributed sequences, for any $R$ we know that
$$ \varlimsup_{n\to\infty} \frac{1}{n} \sum_{k=1}^{n} f(k\beta) \leq \lim_{n\to\infty} \frac{1}{n} \sum_{k=1}^{n} \max\{ f(k\beta), -R \} = \int_{0}^{1} \max\{f(x), -R\} \, dx. $$
Taking $R \to \infty$, we get
$$ \varlimsup_{n\to\infty} \frac{1}{n} \sum_{k=1}^{n} f(k\beta) \leq \int_{0}^{1} f(x) \, dx = 0. \tag{1} $$
In view of (1), if the sequence $(k\beta \text{ mod } 1 : k \geq 1)$ is very evenly distributed over $\Bbb{R}/\Bbb{Z}$, then we even expect equality in (1). For me, it seems to suggest that we need some hard analysis on it.
B. Let $\beta$ be as before, and write $ n\beta = p_n + \epsilon_n$, where $p_n \in \Bbb{Z}$ and $|\epsilon_n| < 1/2$. Also, let us consider only $n$ which are record-breaking indices for $|\epsilon_n|$ in the sense that $|\epsilon_k| > |\epsilon_n|$ for all $k < n$.
We denote $a_n (\beta) = \left| \prod_{k=1}^{n} (1 - e^{2\pi i k \beta} ) \right|$ and then claim the following:
Claim. There exists constants $C_1, C_2 > 0$ such that for all irrational $\beta$ and for all record-breaking $n$, $$ a_n(\beta) \leq C_1 |\epsilon_n| n \exp( C_2 |\epsilon_n| n \log n ). $$
In view of the Dirichlet approximation theorem, we know that $n|\epsilon_n| < 1$. This gives $a_n(\beta) \leq C_1 n^{C_2}$.
If (a subsequence of) $|\epsilon_n|$ decays at least as fast as $1/(n\log n)$ then we have $\liminf_n a_n = 0$. In particular, for any number $\beta$ with irrational measure greater than 2, this is true. Unfortunately, most interesting numbers are either proved or expected to have irrationality measure 2, so this says nothing anything about OP's question.
Now let $\tilde{\beta}_n = p_n / n$. For brevity, we remove the subscripts from $p_n$, $\epsilon_n$ and $\tilde{\beta}_n$ whenever no confusion arises. Then using the relation $|1 - e^{2ix}| = 2|\sin x|$ for any $x \in \Bbb{R}$, we can write
\begin{align*} a_n(\beta) &= 2|\sin (\pi n \beta)| \left| \prod_{k=1}^{n-1} (1 - e^{2\pi i k \tilde{\beta}} ) \right| \left| \prod_{k=1}^{n-1} \frac{\sin(\pi k \beta)}{\sin(\pi k \tilde{\beta})} \right| \\ &= 2n|\sin(\pi \epsilon)| \prod_{k=1}^{n-1} \left| \frac{\sin(\pi k \tilde{\beta} + \pi k \epsilon / n)}{\sin(\pi k \tilde{\beta})} \right| \\ &= 2n|\sin(\pi \epsilon)| \prod_{k=1}^{n-1} \left| \cos(\pi k \epsilon / n) + \cot(\pi k \tilde{\beta}) \sin(\pi k \epsilon / n) \right| \tag{2} \end{align*}
In the second step, the following observation is used.
Lemma 1. If $n$ is record-breaking, then $n$ and $p_n$ are coprime.
Proof. Assume otherwise. Write $n = d\tilde{n}$ and $p_n = d\tilde{p}$ for $d = \gcd(n, p_n) > 1$. Then $\tilde{n}\beta = \tilde{p} + (\epsilon_n / d)$ and thus $|\epsilon_{\tilde{n}}| = |\epsilon_n| / d < |\epsilon_n|$, contradicting the assumption that $n$ is record-breaking. ////
Since $n$ and $p$ are coprime, we know that $\{ p, 2p, \cdots, (n-1)p \} \equiv \{1, 2, \cdots, n-1 \} \pmod n$. In particular, by permuting the indices we have
$$ \prod_{k=1}^{n-1} |1 - e^{2\pi i k p / n}| = \prod_{k=1}^{n-1} |1 - e^{2\pi i k / n}| = n. $$
We also notice that
$$\sum_{k=1}^{n} |\cot(\pi k / n)| = \mathcal{O}(n \log n). \tag{3} $$
This follows from the inequality $\cot x \leq 1/x$ on $(0, \pi/2]$. Combining these observations, we can bound the product term of (2). Indeed, it is clear that $|\sin(\pi k \epsilon / n)| \leq \pi |\epsilon|$ for any $1 \leq k \leq n$. Thus
\begin{align*} \prod_{k=1}^{n-1} \left| \cos(\pi k \epsilon / n) + \cot(\pi k \tilde{\beta}) \sin(\pi k \epsilon / n) \right| &\leq \prod_{k=1}^{n-1} ( 1 + \pi |\epsilon| |\cot(\pi k \tilde{\beta})| ) \\ &\leq \exp \left( \sum_{k=1}^{n-1} \pi |\epsilon| |\cot(\pi k / n)| \right) \\ &\leq e^{C|\epsilon|n \log n}. \end{align*}
Here, we used Lemma 1 to rearrange the product and then used (3) to bound the exponent. This gives the desired upper bound in Claim.
Write it as $a_n(\alpha)$ to emphasize the dependence on $\alpha$. For any $\epsilon > 0$, $U(n,\epsilon) = \{\alpha: |a_n(\alpha)| < \epsilon\}$ is an open set containing $k/m$ for any integers $k,m$ with $k$ even and $1 \le m \le n$. Thus $\cup_{n \ge N} U(n,\epsilon)$ is a dense open set, and $\cap_{m \in \mathbb N} \cap_{N \in \mathbb N} \cup_{n \ge N} U(n, 1/m)$ is a dense $G_\delta$. We conclude that there are uncountably many irrational $\alpha$ such that $\liminf_{n \to \infty} |a_n(\alpha)| = 0$. However, I don't see a way to obtain actual convergence to $0$ for irrational $\alpha$.
