Solving this problem for $d$ days is equivalent to constructing $d-2$ mutually orthogonal $N \times N$ Latin squares, which we'll call $L_3$, $L_4$, ..., $L_d$. So your problem is equivalent to finding the maximum number of $N \times N$ mutually orthogonal magic squares. See OEIS sequence A001438 and the references there.
Here is how to go from MOLS to a seating plan. Label the people by pairs $(i,j)$ with $1 \leq i,j \leq N$. On the first day, people with the same $i$-coordinate eat together. On the second day, people with the same $j$ coordinate eat together. On day $k$, for $k \geq 3$, group people according to the entries of the corresponding cells in $L_k$.
This construction is easily reversible: Given a seating arrangement, number the tables arbitrarily on each day. The $(i,j)$ entry in $L_k$ is given by finding the unique person who ate at table $i$ on day $1$ and table $j$ on day $2$, and writing down the number of the table where they ate on day $k$.
This means that your conference choose uniquely poorly in taking $N=6$! It is possible to achieve $4$ days for every $N$ except $N=2$ and $N=6$. See:
Bose, R. C.; Shrikhande, S. S.; Parker, E. T., Further results on the construction of mutually orthogonal latin squares and the falsity of Euler’s conjecture, Can. J. Math. 12, 189-203 (1960). ZBL0093.31905.
If $N$ is odd, the solution for $4$ days is particularly easy to describe: Seat the people by $i$-coordinate on day $1$, by $j$-coordinate on day $2$, by $i+j$ on day $3$ and by $i+2j$ on day $4$.
More generally, if $p$ is the smallest prime divisor of $N$, then I believe that we can achieve $p+1$ days by using $i$, $j$, $i+j$, $i+2j$, ..., $i+(p-1)j$. I also have a construction which gets $\min(p_i^{e_i})+1$ days, when $N = p_1^{e_1} p_2^{e_2} \cdots p_r^{e_r}$ is the prime factorization of $N$, but I'm going to hold off writing it up.