The number of Dyck paths in a square is wellknown to equal the catalan numbers: http://mathworld.wolfram.com/DyckPath.html But what if, instead of a square, we ask the same question with a rectangle? If one of its sides is a multiple of the other, then again there is a nice formula for the number of paths below the diagonal, but is there a nice formula in general? What is the number of paths from the lowerleft corner of a rectangle with side lengths a and b to its upperright corner staying below the diagonal (except for its endpoint)? I am also interested in asymptotics.

2$\begingroup$ Can you be more precise about the definition you're using here? Should we take "diagonal" to mean diagonal of the rectangle? $\endgroup$ – Qiaochu Yuan Oct 22 '09 at 22:18

$\begingroup$ Yes, the diagonal of the rectangle from its lowerleft vertex to its upperright vertex. $\endgroup$ – domotorp Oct 22 '09 at 23:47

1$\begingroup$ In the time since this question has been asked, there has been a huge amount of interest in "rational Catalan combinatorics." Googling that phrase will give you relevant information; for instance, the slides math.miami.edu/~armstrong/Talks/RCCinDC.pdf and math.umn.edu/~reiner/Talks/AIM2012/AIMIntro.pdf. $\endgroup$ – Sam Hopkins Nov 30 '14 at 4:33

$\begingroup$ @Sam: Thx. In fact my question has been satisfactorily answered by Mirko, we have even written a paper since then: iospress.metapress.com/content/y6u34j013371n571/… $\endgroup$ – domotorp Nov 30 '14 at 12:53
Since that Mirko Visontai told me that the answer is ${a+b\choose a}/(a+b)$ if $\gcd(a,b)=1$. The proof is the following (with k=a and l=b):
The number of 01 vectors with $k$ 0's and $l$ 1's is ${k+l\choose k}$, so we have to prove that out of these vectors exactly $1/(k+l)$ fraction is an element of $L(k,l)$. The set of all vectors can be partitioned into equivalence classes. Two vectors $p$ and $q$ are equivalent if there is a cyclic shift that maps one into the other, i.e., if for some $j$, $p_i = q_{i+j}$ for all $i$. We will prove that exactly one element from each equivalence class will be in $L(k,l)$. This proves the statement as each class consists of $k+l$ elements because $gcd(k,k+l)=1$.
We can view each 01 sequence as a walk on $\mathbb R$ where each 0 is a $l/(k+l)$ step and each 1 is a $+k/(k+l)$ step. Each $(k,l)$ walk starts and ends at zero and each walk reaches its maximum height exactly once, otherwise $ak + bl = 0$ for some $0 < a +b < k+l$ which would imply $\gcd(k,l) \neq 1$. If we take the cyclic shift that ``starts from the top'', we stay in the negative region throughout the walk, which corresponds to remaining under the diagonal in the lattice path case. Any other cyclic shift goes above zero, which corresponds to going above the diagonal at some point.
I heard a talk at Indiana University last March by Timothy Chow. Here's his abstract, which seems to give a negative answer to your question about rectangles whose sides have noninteger ratio:
It is a classical result that if k is a positive integer, then the number of lattice paths from (0,0) to (a+1,b) taking unit north or east steps that avoid touching or crossing the line x = ky is
(a+b choose b)  k (a+b choose b1).
Disappointingly, no such simple formula is known if k is rational but not an integer (although there does exist a determinant formula). We show that if we replace the straightline boundary with a periodic staircase boundary, and if we choose our starting and ending points carefully, then the natural generalization of the above simple formula holds. By varying the boundary slightly we obtain other cases with simple formulas, but it remains somewhat mysterious exactly when a simple formula can be expected. Time permitting, we will also describe some recent related work by Irving and Rattan that provides an alternative proof of some of our results.
This is joint work with Chapman, Khetan, Moulton, and Waters.

$\begingroup$ Odd. I would expect that the generating function is algebraic because the corresponding language should be contextfree, so I'm surprised nobody's worked it out. $\endgroup$ – Qiaochu Yuan Oct 23 '09 at 2:07

3$\begingroup$ Nice, the full paper is here: wwwmath.mit.edu/~tchow/lattice.pdf $\endgroup$ – domotorp Oct 23 '09 at 15:12
Is a sum OK?
I am used to a different rotation of the paths. I think the paths you are looking for can also be described as all paths above the xaxis, with steps (1,1) and (1,1), that starts at (0,0) and ends on the line x=y+n for some (x,y) from (n,0) to (n+m,m).
(If instead they end at the line x=n, we get the Ballot paths.)
Let B(n,k) be the Ballot numbers, B(n,k)= # paths from (0,0) to (n,k). Now, all paths must pass the line x=n. From there on it is just a binomial path, so the number of paths are sum_{k=0,2,4,...,n} B(k,n)*( (nmk)/2 choose k/2)
(n choose k)= Binomial coefficient, n!/(k!(nk)!)

$\begingroup$ I am afraid that your problem with the 1,1 and 1,1 steps seems to be different from the one I described. $\endgroup$ – domotorp Oct 23 '09 at 1:10

$\begingroup$ No? Take the rectangle, rotate it 45 degrees clockwise so that the diagonal becomes the xaxis, then reflect it in the xaxis. $\endgroup$ – Robert Parviainen Oct 23 '09 at 9:49

$\begingroup$ Then the steps would not be 1,1 and 1,1 but something depending on a and b. $\endgroup$ – domotorp Oct 23 '09 at 14:48
If I understood your question correctly, the numbers you're looking for are called Ballot numbers. The number of paths from $(0,0)$ to $(m,n)$ (where $m>n$) which stay below the diagonal is $\frac{mn}{m+n}\binom{m+n}{m}$.
Moreover, if $m>r \cdot n$, then the number of lattice paths from $(0,0)$ to $(m,n)$ which stay below the line $x=r\cdot y$ is $\frac{mrn}{m+n}\binom{m+n}{m}$. (I haven't worked this out, but Ira Gessel says so in Introduction to Lattice Path Enumeration)

$\begingroup$ No, this is not what I want. If the sides of the rectangle are 100 and 250, then the ballot numbers would give an answer for a path that stays above the line from (0,0) to (100,200). Btw, this is why I have written that I also know the answer if one side is a multiple of the other. $\endgroup$ – domotorp Oct 22 '09 at 23:46