Complex Analysis - Analytic Continuation and Residual Integration

At first an example which I know how to treat. Let's say we have the following integral

$\int dx \frac{1}{x^2+a^2}$

Now we do an analytic continuation of the constant $a$ to the complex numbers: $a \rightarrow ia$. This gives

$\int dx \frac{1}{x^2-a^2}$

which now has poles at $x = \pm a$. Let's say I want to evade these poles during the integration (small half-circles into the complex plane around the singular points), then I have to take the half-circle at $x = -a$ into the lower half-plane (integration counter-clockwise), and the half-circle at $x = a$ into the upper half-plane (integration clockwise). As can be seen in this picture: >click<

Whether the integration of a half-circle is clockwise or counter-clockwise, determines the extra sign before the residual part of the integral to be minus or plus respectively.

Now let's get to my actual question. Imagine, instead of the simple example above we have the following integral:

$\int_{0}^{\frac{\pi}{2}} dx \frac{1}{1-f(a^2)\sin^2(x)}$

where $f(a^2) = \frac{\text{polynomial in }a^2}{\text{other polynomial in }a^2}$ and $0 < f(a^2) < 1$ for all $a$. Since $0 < f(a^2) < 1$, there are no singularities in the integral. Now we do the analytic continuation $a \rightarrow ia$. This gives

$\int_{0}^{\frac{\pi}{2}} dx \frac{1}{1-f(-a^2)\sin^2(x)}$

and it is given that always $f(-a^2) > 1$. That causes a singularity to appear in $x = \arcsin(\frac{1}{\sqrt{f(-a^2)}})$. (Before the analytic continuation there was no singularity, because $\arcsin(y)$ is defined only for $-1\leq y\leq 1$). It is not hard to compute the residue at this singularity, but now it is less transparent, whether we should evade the pole drawing the small half-circle into the upper or into the lower half-plane. This little detail determines the sign of the residual part of the integral (I would like to separate it in the residual and principal parts). How can I determine whether I should draw the half-circle in the lower or in the upper half-plane in the general case, or in this situation at least?

-

Notice that in the picture you linked to, the poles moved from $\pm ia$ toward the real axis along counter-clockwise arcs that did not cross the real axis itself (where the integration contour lies). It is this path of approach of the poles toward the real axis that determines how the integration contour needs to be deformed. Informally, the poles make little "dents" in the integration contour as they approach it. If the poles were moved from $\pm ia$ toward the real axis along clockwise arcs, then these dents would be in the other direction and you would get a different sign in front of the residual part of the integral and hence a different answer overall. This shows that the final answer for the integral has branch cuts in its analytic dependence on $a$.
The answer to your main question depends on the way the location of the pole at $x_z=\arcsin(f(z^2)^{-1/2})$ as a function of $z$. You need to explicitly pick a path in the complex $z$-plane that takes you from $a$ to $-ia$ such that the pole $x_z$ does not cross your integration contour. As the pole approaches the real axis, it will make a dent in your integration contour, which will determine the way you need to deform it.
Thank you, your answer is very helpful! I would like to learn more about branch cuts. I thought, the movement of $\pm ia$ toward the real axis in a $counter-clockwise arc$ would be the mathematically conventional way to go about (counter-clockwise as the standard direction). What exactly are these branch cuts? Can they be visualized on the plane for the first simple example? Maybe you can recommend a reference on branch cuts in general? – Fejwin Aug 18 '12 at 20:52
You can read about branch cuts in any book on complex analysis. However, knowing more about branch cuts, other than their existence, will not really help you evaluate the integrals like then ones you discussed above. You just need to keep in mind that the analytic continuation of a function $g(a)$, whether it comes from an integral or not, may depend on the path you take in the complex $a$-plane. This issue is also covered on in any book on complex analysis. – Igor Khavkine Aug 20 '12 at 7:17