A very important role for the undetermined constants in indefinite integrals, in fact perhaps their first really essential role, is in solving basic differential equations. The constants of integration are used to find the solution satisfying the initial conditions for the differential equation. Changing the initial conditions will change the solution, which mathematically corresponds to changing the undetermined constants in indefinite integration. You'd make the whole subject of differential equations more awkward by avoiding the undetermined constants. In fact, it's hard to imagine how anyone could teach or learn basic differential equations without those undetermined constants from integration. (Could the OP write about Fourier transforms but not have studied differential equations?)
It is quite misleading to avoid facing the plain fact that on an interval of the real line, $f'(x) = g'(x) \Longleftrightarrow f(x) = g(x) + C$ for some constant $C$, or even more basically $f'(x) = 0 \Longleftrightarrow f(x) = C$. These undetermined constants in integration are an essential feature of differentiation, just as much as the fact that a system of linear equations $A{\mathbf x} = {\mathbf 0}$ can have a nonzero solution (or, in the language of abstract algebra, that homomorphisms can have nontrivial kernels).
An $n$th order constant coefficient linear differential equation generally has an $n$-dimensional solution space (e.g., $y'' + y = 0$ has solution space $\{a\sin x + b\cos x : a, b \in {\mathbf R}\}$, which is important in physics). These $n$ dimensions, intuitively, come from integrating $n$ times to pass from the differential equation back to its solutions, because each integration introduces an undetermined constant, so an $n$th order differential equation will have $n$ undetermined constants for its solutions (hence an $n$-dimensional solution space). If you want to have an intuition for higher-order differential equations then you want to have the language of undetermined constants available.
Finally, it seems to me that the OP is suggesting (indirectly) that all antiderivatives be fixed by specifying their value at $x = 0$, but the value of most functions at $0$ are no more special or important in general than their value at $x = 1$ or at other numbers, so specifying an indefinite integral by its value at 0 is not in general going to make anything simpler for the purpose of applications. And what would you do for functions like $f(x) = 1/x$, which aren't even defined at $x = 0$? If you want to discuss antiderivatives of functions on an interval that does not contain $0$, it doesn't make sense to specify an antiderivative in terms of its value at $0$.