Let $q = p^t$ where $p$ is prime. I am interested in estimating the complete exponential sum, which looks like $$ \sum_{0 \leq h < q} \chi( (h-a_1)(h-a_2)(h-a_3)) \ \bar{\chi}( (h-b_1)(h-b_2)(h-b_3)) \ e^{ 2 \pi i C h /q} $$ in terms of integers $a_i$'s and $b_j$'s and $C$. Here $\chi$ is a Dirichlet character modulo $q$. I was able to find a reference for something similar but I couldn't quite deduce the bound for this sum. I would greatly appreciate if anyone knows bound for this. Thank you very much.

Alpoge's comments oversimplify the situation slightly.

There exists a unique constant $\alpha$ modulo $p^{\lfloor t/2\rfloor}$ such that $\chi(1+x)=q^{ 2\pi i \alpha x/q}$ for $\alpha$ a mulitple of $p^{\lceil t/2 \rceil}$.

If we let $e_q(x) = e^{2\pi i x /q}$ and $F(h) = \chi( (h-a_1)(h-a_2)(h-a_3)) \ \bar{\chi}( (h-b_1)(h-b_2)(h-b_3)) e_q( C h )$ then $$F(h+x)= F(h) e_q\left( x \left( \alpha \left( \frac{1}{h-a_1} + \frac{1}{h-a_2} + \frac{1}{h-a_3} - \frac{1}{h-b_1} - \frac{1}{h-b_2}- \frac{1}{h-b_3} \right) + C \right) \right).$$

Thus the sum over the residue class vanishes unless$$ \alpha \left( \frac{1}{h-a_1} + \frac{1}{h-a_2} + \frac{1}{h-a_3} - \frac{1}{h-b_1} - \frac{1}{h-b_2}- \frac{1}{h-b_3} \right) + C =0$$ modulo $p^{\lfloor t/2 \rfloor}$.

If this rational function has only simple zeroes then it will have $O(1)$ zeroes in the $t$ even case or $O(p)$ zeroes in the $t$ odd case but with Gauss sum cancellation.

However, in other cases, where the vanishing order is higher, you can get less than square root cancellation for $t$ large. This happens when your sum locally looks like $e_q(x^3)$, for instance.

I have migrated these answers from @alpoge's comments (first comment), and marked it community wiki to avoid reputation. I have done the transcription mostly by hand (cut-and-pasting with TeX code is tough, or at least I don't know how to do it), and have edited lightly, so I have probably introduced errors.

Hey, so let's say $p > 5$ or something. When $t = 1$ this is a standard mixed character sum and can be bounded using Weil's bound (see Weil - On some exponential sums, the last equation on p. 206). Also, unless $\chi$ has conductor $q$, the sum is $0$ (split the sum into fibres modulo $p^{t − 1}$ and you're summing a nontrivial additive character).

Otherwise, write $h \mathbin{=:} x + p k$ with $0 \le x < p$ and $k \in \mathbb Z/p^{t - 1}\mathbb Z$ to obtain $$ \sum_x e_q(x)\prod_i \chi(x - a_i)\overline\chi(x - b_i)\sum_k e_q\Bigl(C p k - A_\chi\sum_{1 \le j \ll t} \frac{(-1)^{j - 1}p^j k^j}j\sum_i (x - a_i)^{-j} - (x - b_i)^{-j}\Bigr), $$ where $A_\chi \in (\mathbb Z/p^{t - 1}\mathbb Z)^\times$. (Here I've written,

viathe $p$-adic logarithm (i.e., $(\mathbb Z/q\mathbb Z)^\times \cong \mathbb F_p^\times \times \mathbb Z/p^{t - 1}\mathbb Z$), $\chi(1 - p t) = e_q\bigl(-A_\chi\sum_{1 \le i \ll t} \frac{p^i t^i}i\bigr)$,i.e., as an additive character on $\mathbb Z/p^{t - 1}\mathbb Z$.)Now, by writing $k \mathbin{=:} m + p^{t - 2}n$ with $0 \le m < p^{t - 2}$ and $n \in \mathbb Z/p\mathbb Z$ and examining the sum over $n$, one sees that, unless $$ A_\chi\sum_i (x - a_i)^{-1} - (x - b_i)^{-1} \equiv C \pmod{p^{t - 1}} $$ (which has $\mathrm O(1)$-many solutions for $x \in \mathbb F_p$), the inner sum is $0$. If $t = 2$ then we're done, and we get a bound of $\mathrm O(1)\mathbin\cdot p = \mathrm O(p)$, since there are $\mathrm O(1)$ $x$ remaining, and, for those $x$, the sum over $k$ is $p$.

Otherwise, when $t \ge 3$ and the linear term does vanish, instead write $k \mathrel{=:} m + p^{\lceil(t - 2)/2\rceil} n$ with $0 \le m < p^{\lceil(t - 2)/2\rceil}$ and $n \in \mathbb Z/p^{\lfloor(t - 2)/2\rfloor}\mathbb Z$, and observe that the sum becomes linear in $n$, with coefficient depending on $m$. Unless

thatcoefficient vanishes, which I think only happens when $m = 0$ (or the $a_i$ and $b_i$ are in remarkable position, but let's ignore that possibility) (forgive me if I'm being sloppy–I figured I'd give a quick answer to this just to help a bit), the sum over $n$ is $0$.

I think that the occurrence of both $\lceil\rceil$ and $\lfloor\rfloor$ in the previous comment may be a mistake, but I'm not sure which was intended.

Otherwise, $x$ has $\mathrm O(1)$ choices, $m = 0$, and the sum over $n$ is $p^{\lfloor(t - 2)/2\rfloor}$, so it seems like you get a bound of $\ll p^{\lfloor(t - 2)/2\rfloor}$ when $t > 2$. Forgive me if I've made a mistake! Just let me know and I'll try to fix it; I admit I'm just writing what comes to mind immediately, so it could be total nonsense. (For example, it seems the exponent of $p$ should have a ceiling or something, since, for $t = 3$, one shouldn't get a bound of $\mathrm O(1)$.)

thatcoefficient vanishes, which I think only happens when $m=0$ (or the $a_i$ and $b_i$ are in remarkable position, but let's ignore that possibility) [forgive me if I'm being sloppy --- I figured I'd give a quick answer to this just to help a bit], the sum over $n$ is $0$. $\endgroup$ – alpoge Feb 24 '18 at 6:35