For $d=q-1$, let $f$ be a polynomial vanishing at all points of $\mathbb F_q$ except $0$.  Let $f_1$ $f_2$ such that $f_1+f_2=f$, and define $A$ such that $A(f_1)=1$ and $A(f_2)=-1$ and $A(g)=0$ for all other $g$.  Then, the only terms surviving are 

$$A(f_1)A(f_2)\phi(f_1+f_2)+A(f_2)A(f_1)\phi(f_2+f_1)+A(f_1)^2\phi(2f_1)+A(f_2)^2\phi(2f_2)$$.

Then, we can compute the value to be 

$$-\phi(f)+-\phi(f)+\phi(0)+\phi(0)=-(q-1)+-(q-1)+q+q=2$$.

Now, I can't prove that this is minimal, but this does show that $q$ isn't minimal for all $d$.

I'll edit this answer if I come up with anything for the $\frac{q}{2}-1$ case.