Since $\{ x: f'(x) \textrm{ does not exist}\}$ is a Borel set, it seems reasonable to restrict ourselves to Borel sets. Then only countable sets work. An uncountable Borel set $X$ has a perfect subset $A$; see here. So $A$ is compact, uncountable, and has no isolated points.

Now we can mimic the construction of the Cantor function: write $A^c$ as the union of its connected components $\bigcup I_n$, pick an $I$ such that $A$ is uncountable both to the left and right of $I$ and set $f_1=1/2$ on $I$ and $f_1$ increases continuously from $0$ to $1$ on $[0,1]$ etc.

Then the $f_n$ converge to a continuous function $f$ with $f'=0$ off $A$, but $f(0)=0$, $f(1)=1$.

Since $\{ x: f'(x) \textrm{ does not exist}\}$ is a Borel set, it seems reasonable to restrict ourselves to Borel sets. Then only countable sets work. An uncountable Borel set $X$ has a perfect subset $A$; see here. So $A$ is compact, uncountable, and has no isolated points. Everything is trivial of course if $X$ contains an interval, so we can also assume that $A$ is nowhere dense (and thus a Cantor set).

Now we can mimic the construction of the Cantor function: write $A^c$ as the union of its connected components $\bigcup I_n$, pick an $I$ such that $A$ is uncountable both to the left and right of $I$ and set $f_1=1/2$ on $I$ and $f_1$ increases continuously from $0$ to $1$ on $[0,1]$ etc.

Then the $f_n$ converge to a continuous function $f$ with $f'=0$ off $A$, but $f(0)=0$, $f(1)=1$.