I don't think there is anything deep going on here. The Fourier analysis on finite abelian group is fairly straightforward.
Gauss sums are the Fourier coefficients you get when you expand an additive character $k \rightarrow e^{\frac{2\pi iak}{p}}$ with respect to the basis of multiplicative characters (i.e. those that give rise to the Dirichlet character characters when our group is $\mathbb{Z}/n\mathbb{Z}$). A Gauss sum is a sum of the product of an additive and a multiplicative characters and as such can be thought of as a finite group analogue of the Gamma function. Recall that the Gamma function is the integral on $\mathbb{R}^{>0}$ of the product of $e^{-x}$ (additive character on the reals) and $x^s$ (a multiplicative character on $\mathbb{R}^{>0}$ \mathbb{R}^{>0}$) with respect to the Haar measure $\frac{dx}{x}$ on $\mathbb{R}^{>0}$.
You are probably thinking ${\zeta_p}^{ak^2}$ as the finite analogue of the Gaussian $e^{-\pi x^2}$, but as you have written yourself,
$g_p(a)=\sum_{k=0}^{p-1} {\zeta_p}^{ak^2}$,
a Gauss sum is a sum of `things' that look like the Gaussian and there is no reason why a Gauss sum itself should be something like the Gaussian.
