I hope this question is reasonable enough to have a well known answer. i.e either there is a simple invariant (like the homotopy groups) that characterizes the homeomorphism type of such set among other such sets or there is an explanation of why such classification is difficult. partial results, like the well known fact that every two open connected and simply connected subsets of the plane are homeomorphic, would be interesting too. thanks.

$\begingroup$ If there is even a sensible homotopy theoretic characterization of the complement of the Cantor set, I'd be curious to know about it. $\endgroup$– Paul SiegelMay 7 '11 at 14:01

1$\begingroup$ @Paul, isn't the complement of the cantor set with respect to the real line just a disjoint union of countably many open intervals like any other open set in R? $\endgroup$– KotelKanimMay 7 '11 at 14:54

$\begingroup$ I just want to make it clear that this question is orthogonal to the question whether there is a criterion for an abstract topological space to be homeomorphic to an open subset of an euclidean space. $\endgroup$– KotelKanimMay 7 '11 at 14:58

5$\begingroup$ Every open subset of the real line is the disjoint union of countably many intervals, is it not? It seems to me that the question is only interesting starting in dimension 2, and indeed my comment was intended to be about the Cantor set sitting in higher dimensional Euclidean space. $\endgroup$– Paul SiegelMay 7 '11 at 16:37

3$\begingroup$ related question: mathoverflow.net/questions/18195/… $\endgroup$– Ian AgolMay 7 '11 at 16:55
If you have a presentation $P$ of a group $G$ with finitely many generators and relations then you can construct a $2$dimensional simplicial complex $K(P)$ with $\pi_1(K(P))=G$. You can then embed $K(P)$ in a Euclidean space (I think $\mathbb{R}^5$ is good enough) and take a regular neighbourhood $U$. Then $U$ is an open subset of $\mathbb{R}^5$ that is homotopy equivalent to $K(P)$ with $\pi_1(U)=G$. The theory of finite group presentations is undecidable, so this means that there is no effective classification of such subsets. This is without any kind of local wildness. You can construct much worse examples by taking the complements of wild knots, fractals, the Alexander horned sphere and so on.

1$\begingroup$ thank you for your answer which indeed shows why such classification can't be effective. It still leaves open the question of a "noneffectiveyettheoreticallyinteresting" classification. For example, whether there are two such sets which are nonhomeomorphic yet with isomorphic homotopy groups (I guess there are). I would like to know if there is some classification of this sort, even if it is not effective, or maybe at least there is in low dimensions and if not, "why" not. thanks again for the answer, I would still like to get some more information on this. $\endgroup$ May 7 '11 at 17:08

2$\begingroup$ @KotelKanim  Starting in dimensions three there are pairs of manifolds that are homotopy equivalent but not homeomorphic. The first examples are lens spaces: en.wikipedia.org/wiki/Lens_space $\endgroup$– Sam NeadMay 9 '11 at 7:46
For subsets of the line, it's easy: each open subset is a countable disjoint union of intervals. For subsets of the plane, It's nontrivial, but something sensible can be said. For subsets of $\mathbb R^n$ for $n\ge 3$, I have no idea and I suspect that it's already hopeless in the case of $\mathbb R^3$.
So let $U\subset \mathbb R^2$ be an open subset, and let's assume without loss of generality that it's connected. Then $U$ is completely classified by its space of ends (see comment of Agol below for a reference). That's a compact, separable, zerodimensional topological space. Those can probably be completely classified, even though I didn't quite manage to do it.
Added later: Actually, there are good reasons why the classification is not feasable (see comments of Clinton Conley below).
The basic operation on such a set $X$ is that of taking its derived subset $$ X':=X\setminus \{\mathrm{isolated\\ points\\ of\\ } X\} $$
This operation can be iterated transfinitely, and so we get $X^{(\alpha)}$ for any ordinal $\alpha$ (where $X^{(\alpha)}$ is defined as the intersection of all $X^{(\beta)}$ for $\beta<\alpha$ when $\alpha$ is a limit ordinal). Since $X^{(\alpha)}$ is a descending sequence of subset of $X$, there is a minimal $\gamma$ such that $X^{(\gamma)}=X^{(\gamma+1)}$. This is a countable ordinal and is the first interesting invariant of $X$. There are two options for $X^{(\gamma)}$: either it's empty, or it's a Cantor set: that's the second invariant of $X$. Finally, if $\gamma$ is a successor ordinal, then you can look at the discrete set $X^{(\gamma1)}\setminus X^{(\gamma)}$. If $X^{(\gamma)}=\emptyset$, then that's a finite nonempty set whose cardinality is an invariant. And if $X^{(\gamma)}$ is a Cantor set, then $X^{(\gamma1)}\setminus X^{(\gamma)}$ is either finite or infinitely countable.
I recapitulate. The invariants of $X:=\mathrm{Ends}(U)$ are:
 The smallest ordinal $\gamma$ such that $X^{(\gamma)}=X^{(\gamma+1)}$.
 Whether or not $X^{(\gamma)}$ is empty or a Cantor set.
 The cardinality of $X^{(\gamma1)}\setminus X^{(\gamma)}$.
I thought for a while that those might be a complete set of invariants of $U$, but I was wrong. For example: if $X^{(\gamma)}$ is a Cantor set, then we can also look at the set of accumulation points of $X^{(\gamma1)}\setminus X^{(\gamma)}$ inside $X^{(\gamma)}$: that's again a compact, separable, zerodimensional topological space, and so it has its own set of invariants...
We can also look at the minimal ordinal $\beta$ such that the closure of $X^{(\beta)}\setminus X^{(\gamma)}$ intersects $X^{(\gamma)}$ nontrivially...
Note: What I attempted to do was the classification of open of subsets $\mathbb R^n$ ($n=1,2$) up to homeomorphism (that was the question, I think), and not up to ambient homeomorphism. The latter is much more messy, already for $n=1$.

2$\begingroup$ This classification follows from results of Richards: ams.org/mathscinetgetitem?mr=143186 $\endgroup$– Ian AgolMay 8 '11 at 0:25

2$\begingroup$ Although, from the point of view of the Borel reducibility hierarchy (where $E \leq_B F$ if there's a Borel function $\phi$ with $x E y$ iff $\phi(x) F \phi(y)$), this doesn't really simplify things much. It is known that homeomorphism of compact subsets of Cantor space sits above every orbit equivalence relation arising from a Borel action of the full symmetric group $S_\infty$ of permutations of the natural numbers. This means that any classification of compacta up to homeomorphism is also a classification of countable structures in a countable language up to isomorphism. To be continued. $\endgroup$ May 8 '11 at 1:41

1$\begingroup$ For instance, you'd be able to identify isomorphic graph structures on the naturals simply by examining the adjacency relation as a subset of $\mathbb{N}^2$. Now that I think about it, if you could definably "reverse the arrows" and associate with each compact subset of Cantor space an open subset of the plane having that designated space of ends (thus reducing the universal $S_\infty$ action to the homeomorphism equivalence relation in question), you'd make a compelling case for the complexity of homeomorphism of open subsets of $\mathbb{R}^2$. Maybe the argument in Richards' paper applies. $\endgroup$ May 8 '11 at 1:50

1$\begingroup$ Actually, it turns out that it's terribly written after all. Unfortunately I don't have the space in this comment to give a reasonable treatment of Borel reducibility among equivalence relations, so I'll cheat and refer you to a paper by Alberto Marcone that does a better job than I did (which you might find otherwise interesting). Complexity of sets and binary relations in continuum theory: a survey, in: Set Theory. Centre de Recerca Matemàtica Barcelona, 20032004, (J. Bagaria and S. Todorcevic, eds.), Trends in Mathematics, Birkhäuser, 2006, pp.121147 To be continued. $\endgroup$ May 8 '11 at 13:47

1$\begingroup$ This is also available on Marcone's web page in the publication list. In particular, pages 46 clarify things, and as a bonus the result for homeomorphism of compact subsets of Cantor space is mentioned at the top of page 6. Part of the problem with "reversing the arrow" as I described is putting a nice Polish structure on a reasonable chunk of open subsets of the plane (I guess one strategy would be fiddling around with representing open sets as countable unions of basic opens, but that seems to have issues). One more comment. $\endgroup$ May 8 '11 at 13:53
For $R^2$, connected open subspaces are precisely the noncompact orientable surfaces with genus zero (see this question for more info.)
For $R^3$, there is no homotopy criterion. The Whitehead manifold embeds in $R^3$ with complement the Whitehead continuum. The Whitehead manifold may be constructed as a union of solid tori, each one homotopically trivial in the next. One may modify the usual construction slightly by embedding each solid torus in the next with a little knot tied in it. Then one can show that this union does not embed in $R^3$. However, such manifolds are contractible, and therefore all homotopy invariants vanish.
To see that it doesn't embed in $R^3$ (or $S^3$), suppose you had an embedding in $S^3$. Then the complement of each solid torus is a knot complement, and each knot is a satellite of the next one. If each such knot were nontrivial, then you would have a knot whose complement contained infinitely many incompressible tori, contradicting Haken finiteness. So infinitely many of these solid tori must have trivial complement (this is exactly what you see in the embedding of the Whitehead manifold). But one may see that the little knot we tie in each embedding of a solid torus in the next must make the resulting knot complement have the little knot as a connect summand, giving a contradiction to embedding in $S^3$.
Here's a descriptive settheoretic view of the problem of classifying open subsets of $\mathbb{R}^2$ up to homeomorphism. The eventual answer will be a formal version of "it's pretty darn hard, but maybe not inconceivably hard." Note that I haven't actually done any work here besides placing the work of others in the correct order.
First, a couple of words on classification problems in general (read: propaganda). When somebody has classified some objects $\mathcal{A}$ in terms of some objects $\mathcal{B}$, two natural questions to ask are
(1) How hard is it to compute the particular $B \in \mathcal{B}$ corresponding to a given $A \in \mathcal{A}$?, and
(2) How hard is it to distinguish elements of $\mathcal{B}$ from each other?
If either of these tasks is particularly hard, one could argue that the classification isn't all that useful. For example, in the current problem one could say "invoke the axiom of choice to wellorder the quotient space of open sets modulo homeomorphism and classify an open set by the location of its equivalence class in the order," which makes (1) rather challenging. Alternatively, one could simply beg the question by saying each open set is in a unique homeomorphismclass, so the homeomorphismclasses themselves provide a classification, which makes (1) easy (in a sense) but there's no reason to think (2) can be done simply.
One way to formalize these concepts is through Borel reducibility of equivalence relations. If $E$ and $F$ are equivalence relations on standard Borel spaces $X$ and $Y$ (resp.), we say $E$ is Borel reducible to $F$, written $E \mathrel{\leq_B} F$, iff there is a Borel function $\phi: X \to Y$ such that $x_0 \mathrel{E} x_1 \iff \phi(x_0) \mathrel{F} \phi(x_1)$. Loosely, the relation $E \mathrel{\leq_B} F$ means "any definable classification of $Y/F$ is also a definable classification of $X/E$" (or, put more succinctly, "$F$ is at least as complicated as $E$"). This gives us a meaningful way to compare various classification problems.(*)
The $\leq_B$ relation is well studied (but there's still a lot we don't know!) in the case that $E$ and $F$ are in themselves in some sense definable. Of particular note is when they are orbit equivalence relations induced by a continuous action of a Polish group on a Polish space. Of even more particular note is when this Polish group is $S_\infty$, the group of permutations of the natural numbers (including those of infinite support). This group is especially important from a logical point of view. Given some countable (let's say relational) language $\mathcal{L}$, you can look at the $\mathcal{L}$structures with universe $\mathbb{N}$. Then the group of automorphisms of $\mathbb{N}$ respecting $\mathcal{L}$ is a closed subgroup of $S_\infty$. In particular, isomorphism of $\mathcal{L}$structures with universe $\mathbb{N}$ is Borel reducible to to an equivalence relation induced by a nice action of $S_\infty$.
Moreover, there's an $S_\infty$universal equivalence relation $E_{S_\infty}$ in the sense that every equivalence relation induced by a nice $S_\infty$ action Borel reduces to $E_{S_\infty}$. In a precise sense, $E_{S_\infty}$ is at least as complicated as the isomorphism relation for any sort of countable structure in a countable language (for example, isomorphism of countable graphs, isomorphism of countable groups, etc.).
Now, the punchline of all this is: there's a Borel reduction of $E_{S_\infty}$ to the homeomorphism relation on open subsets of $\mathbb{R}^2$ (equipped with the natural interpretation of the Effros Borel structure). This follows from
Camerlo and Gao, "The completeness of the isomorphism relation for countable Boolean algebras," Trans AMS 353 (2001), no. 2, 491518
and the the pitchperfect response by André Henriques to a rather nebulous question of mine in the comments after his answer. The idea is that there is a Borel reduction of $E_{S_\infty}$ to homeomorphism of closed subsets of Cantor space (again with the corresponding Effros Borel structure), constructed in CamerloGao. Fixing some copy of Cantor space in the plane, this reduction can then be composed with the Borel map $\phi$ that assigns to each closed subset $K$ of Cantor space its complement in $\mathbb{R}^2$. Since it is known from results of Richards (thanks, Agol!) that $\phi(K_0)$ and $\phi(K_1)$ are homeomorphic iff $K_0$ and $K_1$ are, this composition is indeed a Borel reduction. In fact, if you restrict your attention to nowhere dense subsets of Cantor space you get a Borel reduction to a part of the homeomorphism relation induced by a Polish group action (I'm not sure whether the entire homeomorphism relation is so induced).
So, in other words, any classification of open subsets of $\mathbb{R}^2$ up to homeomorphism is also a classification of any countable structure in any countable language up to isomorphism! So, that's what I meant when I said it's "pretty darn hard."
So then what did "maybe not inconceivably hard" mean? Some equivalence relations are so complicated that they are not classifiable by countable structures. That is, they can not be Borel reduced to any isomorphism relation of $\mathcal{L}$structures as described above. For example, homeomorphism of closed subsets of Cantor space is classifiable by countable structures (Stone duality), while isomorphism of type $II_1$ von Neumann factors is not classifiable by countable structures (SasykTörnquist 2009). I have no idea where this problem sits with respect to this notion (to show that this homeomorphism relation is classifiable by countable structures it would be enough to show that there's a Borel way of associating with each open subset $U$ of the plane a closed subset of Cantor space homeomorphic to $U$'s space of ends). For more examples see:
http://www.math.ucla.edu/~greg/223d.1.09f/countable.pdf
(*)Fine print disclaimer: not everybody thinks this is the "right" hierarchy to use. One alternative is to "effectivize" everything (working lightface rather than boldface: think "recursive" instead of "Borel"), although this is quite challenging and restrictive. But, aside from these, I'm not familiar with any notions with such a rich structure theory. Note that I'm not including the "I'll know a classification when I see it" standpoint (which is fine for SCOTUS, but isn't particularly amenable to formal analysis).
This is an attempt to rephrase the question so that it makes more sense. As mentioned in other answers, there is absolutely no hope for a homeomorphism classification of open subsets of $\mathbb R^n$ if $n>2$.
I think, a better question would be "what can be said about manifolds that are homeomorphic to open subsets of $R^n$, or equivalently, manifolds that can be covered by a single coordinate chart?".
Any such manifold is parallelizable i.e. its tangent bundle is trivial, but I do not know other obstructions. One could consider a parallelizable manifold (e.g. a Lie group) that is not homeomorphic to $R^n$, take any of its open subsets $U$ and then how does one decide whether $U$ can be covered by a single coordinate chart?
So a test question: remove finitely many points from your favorite compact Lie group, and try to see if it embeds in the Euclideas space of the same dimension. Surely, this kind of questions must be answered if one hopes for a classification. I do not know the answer.
Amusingly, any infinite dimensional separable Hilbert manifold is homeomorphic to an open subset of a separable Hilbert space, i.e. $l_2$.

$\begingroup$ There are other obstructions: a torus with a hole is a parallizable open manifold which is not embeddable into the plane. Your restatement is almost surely an open problem  as far as I know there is a work in progress on this subject. $\endgroup$ May 7 '11 at 19:45

$\begingroup$ Regarding "punctured torus does not embed in the plane": sure, because 2dimensional case is special. How does one show that a pinctured ntorus doesn't embed in $R^n$? This must be true, but I have no idea how to even get started. $\endgroup$ May 7 '11 at 19:52

3$\begingroup$ A complete graph $K_5$ can be embedded into the torus $S^1\times S^1$. Thus $K_5\times S^1$ can be embedded into the 3torus $S^1\times S^1\times S^1$. Thus it can be embedded into the punctured 3torus. If the punctured 3torus could be embedded into R^3 then $K_5\times S^1$ would be embedded into R^3. This is a contradiction to known results. If you like the game, search 'Embeding products of graphs' in arXiv. $\endgroup$ May 7 '11 at 20:17

4$\begingroup$ The punctured $n$torus, $n>1$, does not embed in $R^n$. Otherwise, there would be a nonseparating $(n1)$torus in $R^n$, contradicting Alexander duality. $\endgroup$ May 7 '11 at 20:57

4$\begingroup$ It is hard to see how removing more than one isolated point from a connected manifold improves its chances for embedding in codimension 0. Indeed I think one can prove that if $M^n$ is an open $n$manifold and $M^n\text{point}$ embeds in $\mathbb{R}^n$ then so does $M^n$. (A manifold (without boundary) is called open if it contains no compact component.) Consider a codimensionone sphere around the puncture and apply the Schoenflies theorem to cap it off. $\endgroup$ May 7 '11 at 21:05
Open subsets of R^n cannot be classified by their homotopy type only. For instance, take the following two embeddings f,g: S^1 v S^1 v S^2 > R^3. Let f embed both circles S^1 of the wedge to distinct components of R^3S^2, and let g embed them to the same component. Then the open regular neighborhoods of f and g are both homotopy equivalent to S^1 v S^1 v S^2 but they are not homeomorphic to each other because their boundaries are distinct (2 tori  for f, a sphere and a genus 2 surface  for g).
(Nonsense written before removed  thanks to Sergey Melikhov's comment.)

1$\begingroup$ Serezha, thank you very much for your comment. You are absolutely right. The complement to an embedding $S^{p_1}\sqcup\dots\sqcup S^{p_r}\to S^m$ has homotopy type of a wedge only up to dimension m3 (here also $p_1,...,p_r\le m3$). Removed nonsense from the answer. $\endgroup$ May 8 '11 at 7:32

$\begingroup$ Misha, everything looks good now, so I deleted my previous comment (it was not relevant to the original question). $\endgroup$ May 8 '11 at 11:33