At first an example which I know how to treat. Let's say we have the following integral
$\int dx \frac{1}{x^2+a^2}$
Now we do an analytic continuation of the constant $a$ to the complex numbers: $a \rightarrow ia$. This gives
$\int dx \frac{1}{x^2-a^2}$
which now has poles at $x = \pm a$. Let's say I want to evade these poles during the integration (small half-circles into the complex plane around the singular points), then I have to take the half-circle at $x = -a$ into the lower half-plane (integration counter-clockwise), and the half-circle at $x = a$ into the upper half-plane (integration clockwise). As can be seen in this picture: >click<
Whether the integration of a half-circle is clockwise or counter-clockwise, determines the extra sign before the residual part of the integral to be minus or plus respectively.
Now let's get to my actual question. Imagine, instead of the simple example above we have the following integral:
$\int_{0}^{\frac{\pi}{2}} dx \frac{1}{1-f(a^2)\sin^2(x)}$
where $f(a^2) = \frac{\text{polynomial in }a^2}{\text{other polynomial in }a^2}$ and $0 < f(a^2) < 1$ for all $a$. Since $0 < f(a^2) < 1$, there are no singularities in the integral. Now we do the analytic continuation $a \rightarrow ia$. This gives
$\int_{0}^{\frac{\pi}{2}} dx \frac{1}{1-f(-a^2)\sin^2(x)}$
and it is given that always $f(-a^2) > 1$. That causes a singularity to appear in $x = \arcsin(\frac{1}{\sqrt{f(-a^2)}})$. (Before the analytic continuation there was no singularity, because $\arcsin(y) $ is defined only for $-1\leq y\leq 1$). It is not hard to compute the residue at this singularity, but now it is less transparent, whether we should evade the pole drawing the small half-circle into the upper or into the lower half-plane. This little detail determines the sign of the residual part of the integral (I would like to separate it in the residual and principal parts). How can I determine whether I should draw the half-circle in the lower or in the upper half-plane in the general case, or in this situation at least?