Here is a solution for $n=3$.
1. We claim that maximum cardinality of $S$ is $(N-1)N(N+1)/3$. Notice that this is achieved by the set of all words of the form $a_ia_ja_k$ with $i\geq j<k$: this set is self-segregating, and the number of such words with middle letter $a_j$ is $j(j-1)$, so the total number of words is
$$
\sum_{j=1}^N j(j-1)=\frac{(N-1)N(N+1)}3.
$$
It remains to prove the upper bound.
2. Given a self-segregating set $S$, sonsider the following digraph $G$ with $N$ letters $a_1,a_2,\dots,a_N$ as vertices. We draw a red edge $a_i\to a_j$ if $S$ contains a word of the form $a_ia_j*$, and a blue such edge if $S$ contains a word of the form $*a_ia_j$. Denote by $B_0$ the set of vertices having zero blue in-degree, and by $R_0$ the set of vertices having zero red out-degree.
Say that an ordered pair $(a_i,a_j)$ is bad if there are both red and blue edges $a_i\to a_j$. This means that $S$ contains some words $U=xa_ix_j$ and $V=a_ia_jy$. In this case, there should be no word $W$ ending with $x$ (otherwise $U$ would appear between $W$ and $V$), and, similarly, no word starting with $y$; hence in this case $x\in B_0$ and $y\in R_0$ for any such $x$ and $y$.
So, given the graph $G$, all words $a_ia_ja_k$ that may appear in $S$ are determined as follows:
$\bullet$ there are a red edge $a_i\to a_j$ and a blue edge $a_j\to a_k$;
$\bullet$ moreover, if $(a_i,a_j)$ is bad, then $a_k\in R_0$; similarly, if $a_j\to a_k$ is bad, then $a_i\in B_0$.
Surely, we may (and will) assume that every edge in $G$ serves for at least one such word, otherwise we may remove the edge from the graph harmlessly.
Denote the set of such words by $W(G)$. It is now easily seen that, for any $G$, the set $W(G)$ is self-segregating. So we need to bound the maximum cardinality of $W(G)$ over all digraphs $G$.
3. Now we show by induction on $N$ that, for every digraph $F$ on $N$ vertices, we have $|S|\leq (N-1)N(N+1)/3$, where $S=W(G)$. The base case $N=1$ is trivial, as the only word $aaa$ cannot lie in $S$. Now we prerform the step.
Case A. Assume first that $B_0$ is non-empty. Consider any letter $b\in B_0$; we claim that it appears in at most $N(N-1)$ words in $S$, which allows to remove this letter and perform the inductive step.
As $b\in B_0$, the letter $b$ cannot appear at the end of a word in $S$. So any word in $S$ containing $b$ must contain a subword $ba$ with $a\neq b$. For any such $a$, we show that $ba$ appears in at most $N$ words from $S$, thus proving the desired bound. There are two subcases:
(i) If there is at most one edge $b\to a$, then $ba$ may appear in a fiixed position of a word in $S$, so there are indeed at most $N$ such words.
(ii) Otherwose, the pair $(b,a)$ is bad. So all words of the form $xba$ in $S$ have $x\in B_0$, while all words of the form $bay$ have $y\notin B_0$, as $a\to y$ is a blue edge. So $ba$ again appears in at most $N$ words from $S$. This finishes case A.
Case B. Now assume that $B_0$ is empty; in particular, there is no bad pair. In this case, we may assume that any ordered pair of vertices is connected by exactly one edge.
For any letter $a$, denote by $f(a)$ the sum of its red in-degree $r(a)$ and its blue out-degree $b(a)$. Then $a$ appears in $r(a)b(a)\leq \lfloor f(a)^2/4\rfloor$ words from $S$. So it suffices to show that
$$
\sum_a \left\lfloor \frac{f(a)^2}4\right\rfloor\leq \frac{(N-1)N(N+1)}3.
$$
Order the vertices as $a_1,a_2,\dots,a_N$ so that $f(a_1)\geq f(a_2)\geq\dots\geq f(a_N)$. Notice that for every $k=1,2,\dots,N$ we have
$$
\sum_{i=1}^k f(a_i)\leq k^2+2k(N-k)=k(2N-k),
$$
as there can be at most one edge between any ordered pair of vertices. This literally means that the tuple $(2N-1,2N-3,\dots,3,1)$ majorizes $(f(a_1),f(a_2),\dots,f(a_N))$.
Denote
$$
g(x)=\left\lfloor\frac{x+1}2\right\rfloor\left(x-\left\lfloor\frac{x+1}2\right\rfloor\right).
$$
It is readily checked that $g$ is continuous, piecewise-linear, (non-strictly) convex, and $g(k)=\lfloor k^2/4\rfloor$ for all non-negative integer $k$. Therefore, Karamata's inequality yields
$$
\sum_{i=1}^N \left\lfloor \frac{f(a_i)^2}4\right\rfloor=\sum_{i=1}^Ng(f(a_i))
\leq \sum_{i=1}^N g(2N+1-2i)=\frac{(N-1)N(N+1)}3,
$$
as desired.
NB. Surely, the estimates from part 3 are tight on the example showed in part 1.
