This answer fleshes out observations made in comments above. The result below is an analogue of Palfy's theorem for nilpotent primitive groups, as requested.

**Prop**. Let $N<S_n$ be a nilpotent primitive group. Then $n$ is prime and $N$ is cyclic of order $n$.

**Proof**. Since $N$ is nilpotent, a minimal normal subgroup $E$ of $N$ is elementary-abelian of order $p^a$. Since $N$ is primitive it must be transitive and, since $E$ is abelian, it must act regularly - so $n=p^a$. Indeed $E$ must be the unique minimal normal subgroup of $N$ because if there another, $E'$ say, then $E'$ would centralize $E$ and $EE'$ would be a transitive abelian subgroup of $S_n$ of order greater than $n$, which is impossible.

Now, since $E$ is unique, we conclude that $C_N(E)=E$. In particular if $g$ is an element of order coprime to $p$, then $g\not\in C_N(E)$. But this contradicts the fact that $N$ is nilpotent. Thus $N$ is a $p$-group. Then $N$ acts on $E$ via linear transformations and so $N/E$ is a $p$-subgroup of $GL_a(p)$ and, in particular, fixes a 1-dimensional subspace of $E$. This subspace is normal in $N$ and hence, since $N$ is primitive, is transitive. i.e. $E$ is itself 1-dimensional, i.e. $E=C_p$ with $n=p$. But, since $N$ is a $p$-group inside $S_p$ we conclude that $N=E=C_p$ as required. **QED**

In fact, rather than primitivity, all I've used here is that $N$ has no intransitive normal subgroups. This property is called *quasiprimitivity* - it is a little weaker than primitivity.

There is an interesting sort of strong converse to this result which also sheds light on the original question.

**Prop**. Let $n$ be a prime and $N<S_n$ be a nilpotent transitive group. Then $N$ is cyclic of order $n$.

**Proof**. Since $N$ is transitive it contains a cyclic subgroup $C$ of order $n$. But $C_{S_n}(C)=C$ and so $N$ must be an $n$-group. But then $N=C$, as required. **QED**

This result, along with the example given above in the comments - a Sylow $2$-subgroup when $n=2^k$ - demonstrate that bounding the order of a nilpotent transitive group is strongly dependent on the prime factorization of $n$. It's not clear to me whether there is a natural stronger condition than transitivity that will hold for any $n$...