Extended binomial coefficients and the gamma function

Question

For which $(a,b,n) \in \mathbb{Z}^3$ satisfying $a+b=n$ does $\frac{\Gamma(z+1)}{\Gamma(x+1)\Gamma(y+1)}$ approach a limit as $(x,y,z) \rightarrow (a,b,n)$ in $\mathbb{C}^3$, and what is that limit? (The case where $a$, $b$, and $n$ are all negative is the problematic one.) What if we restrict to $\{(x,y,z) \in \mathbb{C}^3 \ | \ x+y=z\}$?

I'm guessing that where the limit exists it agrees with the "windmill" shown in https://i.sstatic.net/osFsj.png (taken from page 197 of Hilton, Holton, and Pedersen's "Mathematical Reflections: In a Room with Many Mirrors").

Answer 1

There's nothing special about the gamma function; the failure of the limit to exist when $a$, $b$, and $n$ are negative is exactly the same as the failure of $$\lim_{(x,y,z)\to(0,0,0)} \frac{xy}{z}$$ to exist.

I will renormalize in a way that I think makes clearer what's going on. First let's look at $$\lim_{(x,y,z)\to(0,0,0)} \frac{\Gamma(z)}{\Gamma(x)\Gamma(y)}.$$ We have $$\frac{\Gamma(z)}{\Gamma(x)\Gamma(y)} = \frac{xy}{z}\frac{\Gamma(z+1)}{\Gamma(x+1)\Gamma(y+1)}.$$ Since $\Gamma(z+1)/\Gamma(x+1)\Gamma(y+1)$ is analytic and nonzero near $(0,0,0)$, $\Gamma(z)/\Gamma(x)\Gamma(y)$ behaves just like $xy/z$ near $(0,0,0)$. In particular, if you restrict to $x+y=z$, this is $xy/(x+y)$, for which the limit as $(x,y)\to(0,0)$ does not exist.

Now let's consider $$\lim_{(x,y,z)\to(0,0,0)} \frac{\Gamma(z-a)}{\Gamma(x-b)\Gamma(y-c)}$$ where $a$, $b$, and $c$ are nonnegative integers. We have $$\frac{\Gamma(z-a)}{\Gamma(x-b)\Gamma(y-c)} = \frac{\Gamma(z)}{\Gamma(x)\Gamma(y)}\cdot \frac{(x-b)\cdots (x-1)(y-c)\cdots (y-1)}{(z-a)\cdots(z-1)}.$$ The second factor on the right is analytic and nonzero near $(0,0,0)$, so $\Gamma(z-a)/\Gamma(x-b)\Gamma(y-c)$ behaves just like $\Gamma(z)/\Gamma(x)\Gamma(y)$ (and thus like $xy/z$) near $(0,0,0)$.

Answer 2

Note that, for integer $m \ge 0$, we have poles in $\Gamma$ with known residues: $$ \Gamma(-m+t) = \frac{(-1)^m}{(-m)!\,t} + O(1)\,\qquad\text{as } t \to 0\text{ in }\mathbb C $$ Now suppose we are given integers $a,b,n \le 0$. Now as $t \to 0$ in $\mathbb C$ we have \begin{align} \Gamma(a+t) &= \frac{(-1)^a}{(-a)!\,t}+O(1) \\ \Gamma(b+t) &= \frac{(-1)^b}{(-b)!\,t}+O(1) \\ \Gamma(n+t) &= \frac{(-1)^n}{(-n)!\,t}+O(1) \\ \frac{\Gamma(n+t)}{\Gamma(a+t)\Gamma(b+t)} &= \frac{(-1)^{n-a-b}(-a)!(-b)!t}{(-n)!}+O(t^2) \\ \lim_{t \to 0}\frac{\Gamma(n+t)}{\Gamma(a+t)\Gamma(b+t)} &= 0. \end{align} But also \begin{align} \Gamma(a+t) &= \frac{(-1)^a}{(-a)!\,t}+O(1) \\ \Gamma(b+t) &= \frac{(-1)^b}{(-b)!\,t}+O(1) \\ \Gamma(n+t^3) &= \frac{(-1)^n}{(-n)!\,t^3}+O(1) \\ \frac{\Gamma(n+t^3)}{\Gamma(a+t)\Gamma(b+t)} &= \frac{(-1)^{n-a-b}(-a)!(-b)!}{(-n)!t}+O(1) \\ \lim_{t \to 0}\frac{\Gamma(n+t^3)}{\Gamma(a+t)\Gamma(b+t)} &= \infty. \end{align} So we conclude that $$ \lim_{(x,y,z)\to (a,b,n)} \frac{\Gamma(z)}{\Gamma(x)\Gamma(y)} $$ does not exist.

---

Now consider the case $x+y=z$. Again as $t \to 0$ in $\mathbb C$ we have

\begin{align} \Gamma(a+t) &= \frac{(-1)^a}{(-a)!\,t}+O(1) \\ \Gamma(b+t) &= \frac{(-1)^b}{(-b)!\,t}+O(1) \\ \Gamma(n+2t) &= \frac{(-1)^n}{(-n)!\,2t}+O(1) \\ \frac{\Gamma(n+2t)}{\Gamma(a+t)\Gamma(b+t)} &= \frac{(-1)^{n-a-b}(-a)!(-b)!t}{2(-n)!}+O(t^2) \\ \lim_{t \to 0}\frac{\Gamma(n+2t)}{\Gamma(a+t)\Gamma(b+t)} &= 0. \end{align} But \begin{align} \Gamma(a+t) &= \frac{(-1)^a}{(-a)!\,t}+O(1) \\ \Gamma(b-t+t^3) &= \frac{-(-1)^b}{(-b)!\,t}+O(1) \\ \Gamma(n+t^3) &= \frac{(-1)^n}{(-n)!\,t^3}+O(1) \\ \frac{\Gamma(n+t^3)}{\Gamma(a+t)\Gamma(b-t+t^3)} &= \frac{-(-1)^{n-a-b}(-a)!(-b)!}{(-n)!\,t}+O(t) \\ \frac{\Gamma(n+t^3)}{\Gamma(a+t)\Gamma(b-t+t^3)} &= \infty. \end{align} And thus $$ \lim_{(x,y) \to (a,b)}\frac{\Gamma(x+y)}{\Gamma(x)\Gamma(y)} $$ does not exist.

---

What about the edited version with $\Gamma(z+1)/(\Gamma(x+1)\Gamma(y+1))$? For the first part, we needed only $a,b,n \le 0$ so translating all by $1$ will apply to the edited version.
For the second part, if we have $x+y=z$ we can write $$ \frac{\Gamma(x+y+1)}{\Gamma(x+1)\Gamma(y+1)} = \frac{x+y}{xy}\;\frac{\Gamma(x+y)}{\Gamma(x)\Gamma(y)} $$ and apply the case that I did here. Of course we must note that $x,y,x+y$ are not $0$ and not $\infty$.