MathOverflow will be down for maintenance for approximately 3 hours, starting Monday evening (06/24/2013) at approximately 9:00 PM Eastern time (UTC-4).

Well, not always, e.g. take $X=\mathbb P^2$ with homogeneous co-ordinates $x,y,z$, $U$ the locus defined by $xyz\ne 0$ and $\sigma$ the involution of $U$ defined by $(x,y,z)\mapsto (yz,xz,xy)$ (the standard quadratic Cremona transformation of $X$). Then $\sigma$ is of order $2$ and does not extend to $X$. On the other hand some sufficient conditions, in characteristic zero, are: there are no rational curves in $X\setminus U$; $X$ has ample canonical class and is smooth (this can be weakened to having canonical singularities).

[Update: Charles Siegel asked for references, and I have none to hand, although this is all well known.]

For the Cremona example, recall that every automorphism of $\mathbb P^2$ is linear, and $\sigma$ clearly isn't. More geometrically, $\sigma$ blows up the vertices of the triangle $xyz =0$ and collapses its sides.

No rational curves in $X\setminus U$: fix $g\in G$ and think of it as a rational map $g:X\to X$. By Hironaka, there is a minimal composite $p:Z\to X$ of blow-ups such that $g\circ p$ is regular. If $X\setminus U$ has no rational curves, then the rational curves in the exceptional divisor of the last blow-up are contracted, so the last blow-up was redundant, contradicting minimality.

Ample canonical class $\omega_X$: then $X=Proj(R(X,\omega_X))$, with $R(X)= \oplus_{n\ge 0}H^0(X,\omega_X^{\otimes n})$, the canonical ring. This is a birational invariant of $X$, so $G$ acting on $U$ acts on $R(X)$, so on $X$.

1

Well, not always, e.g. take $X=\mathbb P^2$ with homogeneous co-ordinates $x,y,z$, $U$ the locus defined by $xyz\ne 0$ and $\sigma$ the involution of $U$ defined by $(x,y,z)\mapsto (yz,xz,xy)$ (the standard quadratic Cremona transformation of $X$). Then $\sigma$ is of order $2$ and does not extend to $X$. On the other hand some sufficient conditions, in characteristic zero, are: there are no rational curves in $X\setminus U$; $X$ has ample canonical class and is smooth (this can be weakened to having canonical singularities).