I think I can prove that $f(n) \le n(n-1)$ with equality is and only if $n$ is prime, but it depends on the following result, for which I have not properly worked out the proof.

Let $H$ be the holomorph of the cyclic group $C_n$ of order $n$ - that is the semidirect product of $C_n$ with its automorphism group. Then the maximum order of elements in $H$ is $n$. I have checked by computer that it is true for $n \le 1200$, and I might try and work out the details later.

Anyway, let's assume that is true. If your normal subgroup $M = \langle a \rangle$ is transitive, then $M$ is self-centralizing in $S_n$, so we get $|R| \le n\phi(n)$.

Otherwise, suppose that $M$ has $n/m$ orbits of length $m$ with $m < n$. Then $b$ must act as a cycle of length $n/m$ on the orbits of $M$, so $c := b^{n/m}$ fixes each orbit of $M$. Furthermore, the actions of $b'$ on these orbits are all equivalent, and so the the restriction of $b'$ to each orbit has the same order. Since this restriction lies in the normalizer in $S_m$ of a cyclic subgroup of order $m$, it lies in the holomorph of $C_m$ and so by the (claimed) result above, this order is at most $m$. So the order of $b$ is at most $n$ and hence $|R| \le mn \le n^2/2$.

Note that, for $n = 2m$ with $m$ odd, there are examples of order $n^2/2$.

I think I can prove that $f(n) \le n(n-1)$ with equality if and only if $n$ is prime, but it depends on the following result, for which I have not properly worked out the proof.

Let $H$ be the holomorph of the cyclic group $C_n$ of order $n$ - that is the semidirect product of $C_n$ with its automorphism group. Then the maximum order of elements in $H$ is $n$. I have checked by computer that it is true for $n \le 1200$, and I might try and work out the details later.

Anyway, let's assume that is true. If your normal subgroup $M = \langle a \rangle$ is transitive, then $M$ is self-centralizing in $S_n$, so we get $|R| \le n\phi(n)$.

Otherwise, suppose that $M$ has $n/m$ orbits of length $m$ with $m < n$. Then $b$ must act as a cycle of length $n/m$ on the orbits of $M$, so $c := b^{n/m}$ fixes each orbit of $M$. Furthermore, the actions of $c$ on these orbits are all equivalent, and so the the restriction of $c$ to each orbit has the same order. Since this restriction lies in the normalizer in $S_m$ of a cyclic subgroup of order $m$, it lies in the holomorph of $C_m$ and so by the (claimed) result above, this order is at most $m$. So the order of $b$ is at most $n$ and hence $|R| \le mn \le n^2/2$.

Note that, for $n = 2m$ with $m$ odd, there are examples of order $n^2/2$.