For $N=5$ and more generally $N$ relatively prime to $6$, it is easy to make $N$ rounds.
Name the players $(i,j)$ for $0 \leq i \leq 3$ and $0 \leq j \leq N-1$. In the $a$-th round, take the elements of the $b$-th quadruple to be $\{ (0,b \bmod N), (1,a+b \bmod N), (2, 2a+b \bmod N), (3, 3a+b \bmod N) \}$.
Suppose, to the contrary that $(i_1, j_1)$ and $(i_2, j_2)$ play together in both rounds $a$ and $a'$. Since $(i,j_1)$ and $(i,j_2)$ never play in the same round, we must have $i_1 \neq i_2$. Then the equation that $i_1$ and $i_2$ play together in round $a$ says that $j_1 - a i_1 \equiv j_2 - a i_2 \equiv N$ and, likewise, $j_1 - a' i_1 \equiv j_2 - a' i_2 \bmod N$. So $(a-a') (i_1 - i_2) \equiv 0 \bmod N$. But $i_1 - i_2 \in \{ \pm 1, \pm 2, \pm 3 \}$ and $\text{GCD}(N,6)=1$, so this implies that $a=a'$. $\square$
A more pictorial description. I'll use bridge instead of tennis, because the 4-sides of a table make for good nomenclature. Arrange $N$ bridge tables in a cricle. At each of $N$ tables, let one player sit at north, one at east, one at south and one at north.
After the first set of games, the north players stay put, the east players move on one table, the south players move on two tables and the west players move on three table, each staying in their same position (north, east, west or south). Repeat this $N$ times.