Take the 2-minute tour ×
MathOverflow is a question and answer site for professional mathematicians. It's 100% free, no registration required.

The number of Dyck paths in a square is well-known to equal the catalan numbers: http://mathworld.wolfram.com/DyckPath.html But what if, instead of a square, we ask the same question with a rectangle? If one of its sides is a multiple of the other, then again there is a nice formula for the number of paths below the diagonal, but is there a nice formula in general? What is the number of paths from the lower-left corner of a rectangle with side lengths a and b to its upper-right corner staying below the diagonal (except for its endpoint)? I am also interested in asymptotics.

share|improve this question
Can you be more precise about the definition you're using here? Should we take "diagonal" to mean diagonal of the rectangle? –  Qiaochu Yuan Oct 22 '09 at 22:18
Yes, the diagonal of the rectangle from its lower-left vertex to its upper-right vertex. –  domotorp Oct 22 '09 at 23:47
In the time since this question has been asked, there has been a huge amount of interest in "rational Catalan combinatorics." Googling that phrase will give you relevant information; for instance, the slides math.miami.edu/~armstrong/Talks/RCCinDC.pdf and math.umn.edu/~reiner/Talks/AIM2012/AIMIntro.pdf. –  Sam Hopkins Nov 30 at 4:33
@Sam: Thx. In fact my question has been satisfactorily answered by Mirko, we have even written a paper since then: iospress.metapress.com/content/y6u34j013371n571/… –  domotorp Nov 30 at 12:53

4 Answers 4

up vote 8 down vote accepted

Since that Mirko Visontai told me that the answer is ${a+b\choose a}/(a+b)$ if $\gcd(a,b)=1$. The proof is the following (with k=a and l=b):

The number of 0--1 vectors with $k$ 0's and $l$ 1's is ${k+l\choose k}$, so we have to prove that out of these vectors exactly $1/(k+l)$ fraction is an element of $L(k,l)$. The set of all vectors can be partitioned into equivalence classes. Two vectors $p$ and $q$ are equivalent if there is a cyclic shift that maps one into the other, i.e., if for some $j$, $p_i = q_{i+j}$ for all $i$. We will prove that exactly one element from each equivalence class will be in $L(k,l)$. This proves the statement as each class consists of $k+l$ elements because $gcd(k,k+l)=1$.

We can view each 0--1 sequence as a walk on $\mathbb R$ where each 0 is a $-l/(k+l)$ step and each 1 is a $+k/(k+l)$ step. Each $(k,l)$ walk starts and ends at zero and each walk reaches its maximum height exactly once, otherwise $ak + bl = 0$ for some $0 < a +b < k+l$ which would imply $\gcd(k,l) \neq 1$. If we take the cyclic shift that ``starts from the top'', we stay in the negative region throughout the walk, which corresponds to remaining under the diagonal in the lattice path case. Any other cyclic shift goes above zero, which corresponds to going above the diagonal at some point.

share|improve this answer

I heard a talk at Indiana University last March by Timothy Chow. Here's his abstract, which seems to give a negative answer to your question about rectangles whose sides have non-integer ratio:

It is a classical result that if k is a positive integer, then the number of lattice paths from (0,0) to (a+1,b) taking unit north or east steps that avoid touching or crossing the line x = ky is

(a+b choose b) - k (a+b choose b-1).

Disappointingly, no such simple formula is known if k is rational but not an integer (although there does exist a determinant formula). We show that if we replace the straight-line boundary with a periodic staircase boundary, and if we choose our starting and ending points carefully, then the natural generalization of the above simple formula holds. By varying the boundary slightly we obtain other cases with simple formulas, but it remains somewhat mysterious exactly when a simple formula can be expected. Time permitting, we will also describe some recent related work by Irving and Rattan that provides an alternative proof of some of our results.

This is joint work with Chapman, Khetan, Moulton, and Waters.

share|improve this answer
Odd. I would expect that the generating function is algebraic because the corresponding language should be context-free, so I'm surprised nobody's worked it out. –  Qiaochu Yuan Oct 23 '09 at 2:07
Nice, the full paper is here: www-math.mit.edu/~tchow/lattice.pdf –  domotorp Oct 23 '09 at 15:12

Is a sum OK?

I am used to a different rotation of the paths. I think the paths you are looking for can also be described as all paths above the x-axis, with steps (1,1) and (1,-1), that starts at (0,0) and ends on the line x=y+n for some (x,y) from (n,0) to (n+m,m).

(If instead they end at the line x=n, we get the Ballot paths.)

Let B(n,k) be the Ballot numbers, B(n,k)= # paths from (0,0) to (n,k). Now, all paths must pass the line x=n. From there on it is just a binomial path, so the number of paths are sum_{k=0,2,4,...,n} B(k,n)*( (n-m-k)/2 choose k/2)

(n choose k)= Binomial coefficient, n!/(k!(n-k)!)

share|improve this answer
I am afraid that your problem with the 1,1 and 1,-1 steps seems to be different from the one I described. –  domotorp Oct 23 '09 at 1:10
No? Take the rectangle, rotate it 45 degrees clockwise so that the diagonal becomes the x-axis, then reflect it in the x-axis. –  Robert Parviainen Oct 23 '09 at 9:49
Then the steps would not be 1,1 and 1,-1 but something depending on a and b. –  domotorp Oct 23 '09 at 14:48

If I understood your question correctly, the numbers you're looking for are called Ballot numbers. The number of paths from $(0,0)$ to $(m,n)$ (where $m>n$) which stay below the diagonal is $\frac{m-n}{m+n}\binom{m+n}{m}$.

Moreover, if $m>r \cdot n$, then the number of lattice paths from $(0,0)$ to $(m,n)$ which stay below the line $x=r\cdot y$ is $\frac{m-rn}{m+n}\binom{m+n}{m}$. (I haven't worked this out, but Ira Gessel says so in Introduction to Lattice Path Enumeration)

share|improve this answer
No, this is not what I want. If the sides of the rectangle are 100 and 250, then the ballot numbers would give an answer for a path that stays above the line from (0,0) to (100,200). Btw, this is why I have written that I also know the answer if one side is a multiple of the other. –  domotorp Oct 22 '09 at 23:46

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.