I think the most obvious way to see the symmetric group appearing is the following. To calculate the difference between $f(x)$ and the constant $f(0)$:
$f(x) - f(0) = \int_0^x f'(s_1) ds_1$$$f(x) - f(0) = \int_0^x f'(s_1) ds_1.$$
Often we do this move because $x$ is small, so we figure that a small change in the input to $f$ will result in the output changing depending on the derivative of $f$. But the same logic goes for $f'$, so applying the same formula to $f'$, we get:
$f(x) = f(0) + f'(0) x + \int_{0 < s_2 < s_1 < x} f''(s_2) ds_2 ds_1$$$f(x) = f(0) + f'(0) x + \int_{0 < s_2 < s_1 < x} f''(s_2) ds_2 ds_1.$$
Note $f'$ may not change significantly from its initial value if $f''$ is under control, but there can be oscillations at the level of the next derivative responsible for $f'$ experiencing a small change --— in this case Taylor expansion usually is not helping us understand the problem. In any case, here we have an integral over the region $ \{ 0 < s_2 < s_1 < x \} $. This is a fundamental domain for the action of $S_2$ on the square $\{ 0 < s_1, s_2 < x\}$, so if $f''$ is a constant $f''(0)$, you get the volume $x^2 / 2!$ times $f''(0)$.
Even if $f''$ is not constant, you can certainly write $f''(s_2) = f''(0) + \int_0^{s_2} f''(s_3) ds_3 $ and get $f''(0) x^2 / 2!$ plus an integral over $\{ 0 < s_3 < s_2 < s_1 < x \}$, which is again a fundamental domain for the action of $S_3$ therefore has volume $x^3/3!$. Similar considerations apply to the higher order terms.
Normally we replace the use of Fubini's theorem here with an equivalent integration by parts so that we never see iterated integrals. (Fubini's theorem is one way to prove integration by parts, so it really is the same move.)
