I have seen that if $G$ is a finite group and $H$ is a proper subgroup of $G$ with finite index then $ G \neq \bigcup\limits_{g \in G} gHg^{1}$. Does this remain true for the infinite case also?
Not in general. Every matrix in $\text{GL}_2(\mathbf C)$ is conjugate to an invertible upper triangular matrix (use eigenvectors), and the invertible upper triangular matrices are a proper subgroup.

1$\begingroup$ I suppose that the OP was talking about discrete subgroups, because every element of a compact Lie group is conjugate to some maximal torus, so this sort of "infinite" is out. $\endgroup$ – José FigueroaO'Farrill Jul 31 '10 at 18:21

$\begingroup$ For a discrete (i.e., countable) example consider the example of Osin mentioned in mathoverflow.net/questions/29605/… $\endgroup$ – Torsten Ekedahl Jul 31 '10 at 18:47

3$\begingroup$ For a countable example one could just replace Keith's $\mathbb{C}$ by a countable algebraically closed field. $\endgroup$ – Robin Chapman Jul 31 '10 at 18:53

11$\begingroup$ A quibble: I think this is a perfectly good "discrete" example, i.e., it uses $GL_2(\mathbb{C})$ as a group, not as a topological group. As for using "discrete" as a synonym in group theory for countable, I say boo. Consider for instance the group $\mathbb{Z}$ topologized as a subgroup of $\widehat{\mathbb{Z}}$. $\endgroup$ – Pete L. Clark Jul 31 '10 at 22:29

2$\begingroup$ Osin's example is finitely generated. Surely that's the relevant 'discreteness' criterion? (And if you think that's a minor point, try constructing a finitely presented example!) $\endgroup$ – HJRW Oct 6 '10 at 15:30
In your original question you require $H$ to have finite index in $G$. Most of the other answers are treating this as unintended. If you actually did want to require this, then the result is true for infinite groups as well.
Proof: Let $[G:H]=n$. The action of $G$ on $G/H$ gives a map $G \to S_n$; let $K$ be the kernel of this map. Then $H/K$ and $G/K$ are finite groups, so we know $G/K \neq \bigcup g (H/K) g^{1}$. In particular, there is some coset $fK$ which is not in any $g H g^{1}$. So $f$ is not in any $g H g^{1}$.
If you really want an example with a discrete (i.e. "almostfinite") feel, consider $G=$ union of all symmetric groups $S_n$ acting on $\{1,2,3,\dots\}$ and $H$ the stabilizer of $1$. Since every element of $G$ belongs to some $S_n$ it can be conjugated inside $S_{n+1}$ to an element of $H$. The usual swindle...
(Let me drop the finite index requirement as in the other answers)
This remains true for discrete virtually solvable groups. Indeed the property of having no proper subgroup containing a conjugate of every element in the group is stable by extension.
I let you check that if the property holds for G_1 and G_2, and G is such that there is a sequence 0>G_1>G>G_2>0, then the property also holds for G.
Note that this property is not stable by direct limit. If K is the algebraic closure of a finite field, SL_n(K) is the direct limit of finite groups, yet the property is not verified. This gives an example of an amenable group for which the property does not hold.
It is not yet known if there are amenable groups of finite type which do not satisfy the property.
On the other hand, the property does not hold for nonabelian free groups, or, in the realm of uncountable groups, for connected semisimple complex lie groups that are not solvable.
Given an arbitrary countable group $H$ containing an element of large enough order but no involutions, there exists a twogenerator simple group $G$ such that $H$ is a proper subgroup of $G$ and $G=\bigcup \limits _{g\in G} gHg^{1}$. This is Theorem 17 of
S.V. Ivanov and A.Y. Ol'shanskii, Some applications of graded diagrams in combinatorial group theory, Groups—St. Andrews 1989, Vol. 2, London Math. Soc. Lecture Note Ser., vol. 160, Cambridge Univ. Press, Cambridge, 1991, pp. 258–308.
$gHg^{−1}$
is intended (and "infinite index case"). The natural starting point is to ask whether the proof for finite index breaks down. – $\endgroup$ – Jim Humphreys Jul 31 '10 at 18:20