2 Corrected misstatement of theorem of Smale

As others have noted, it's hopeless to try to answer this question for general topological spaces. However, there are a few positive results if you assume, say, that X and Y are both simply connected closed manifolds of a given dimension. For example, Freedman showed that if X and Y are oriented and have dimension four, then to check whether they're homeomorphic you just need to compute (i) the bilinear "intersection" forms on H^2(X;Z) and H^2(Y;Z) induced by the cup product; and (ii) a Z/2-valued invariant called the Kirby-Siebenmann invariant. The invariant in (ii) obstructs the existence of a smooth structure, so if you happened to know that both X and Y were smooth manifolds (hence that their Kirby-Siebenmann invariants vanished) you'd just have to look at their intersection forms to determine whether they're homeomorphic (however a great many examples show that this wouldn't suffice to show that they're diffeomorphic).

In higher dimensions, Smale showed Smale's h-cobordism theorem shows that two simply connected smooth manifolds are diffeomorphic as soon as there is a homotopy equivalence cobordism between them for which pulls back the tangent bundle on one to the tangent bundle inclusion of the otherboth manifolds is a homotopy equivalence. Of course, Checking this begs the question of how one knows that such a map existscriterion can still be subtle, but work of Wall and Barden shows that in the simply-connected 5-dimensional case it suffices to check that there's an isomorphism on second homology H2 which preserves both (i) the second Stiefel-Whitney classes, and (ii) a certain "linking form" on the torsion subgroup of H2.

If you drop the simply-connected assumption, things get rather harder--indeed if n>3 then any finitely presented group is the fundamental group of a closed n-manifold (which can be constructed in a canonical way given a presentation), and Markov (son of the probabilist) showed that the impossibility of algorithmically distinguishing whether two presentations yield the same group translates to the impossibility of algorithmically classifying manifolds. Even assuming you already knew the fundamental groups were isomorphic, there are still complications beyond what happens in the simply-connected case, but these can sometimes be overcome with the s-cobordism theorem.

In a somewhat different direction, in dimension 3 one can represent manifolds by link diagrams, and Kirby showed that two such manifolds are diffeomorphic (which in dimension 3 is equivalent to homeomorphic) iff you can get from one diagram to the other by a sequence of moves of a certain kind. (see Kirby calculus in Wikipedia; similar statements exist in dimension 4). I suppose that one could argue that this isn't an example of what you were looking for, since if one felt like it one could extract diffeomorphisms from the moves in a fairly explicit way, and one can't (AFAIK) just directly extract some invariants from the diagrams which completely determine whether the moves exist.

1

As others have noted, it's hopeless to try to answer this question for general topological spaces. However, there are a few positive results if you assume, say, that X and Y are both simply connected closed manifolds of a given dimension. For example, Freedman showed that if X and Y are oriented and have dimension four, then to check whether they're homeomorphic you just need to compute (i) the bilinear "intersection" forms on H^2(X;Z) and H^2(Y;Z) induced by the cup product; and (ii) a Z/2-valued invariant called the Kirby-Siebenmann invariant. The invariant in (ii) obstructs the existence of a smooth structure, so if you happened to know that both X and Y were smooth manifolds (hence that their Kirby-Siebenmann invariants vanished) you'd just have to look at their intersection forms to determine whether they're homeomorphic (however a great many examples show that this wouldn't suffice to show that they're diffeomorphic).

In higher dimensions, Smale showed that two smooth manifolds are diffeomorphic as soon as there is a homotopy equivalence between them which pulls back the tangent bundle on one to the tangent bundle of the other. Of course, this begs the question of how one knows that such a map exists, but work of Wall and Barden shows that in the simply-connected 5-dimensional case it suffices to check that there's an isomorphism on second homology H2 which preserves both (i) the second Stiefel-Whitney classes, and (ii) a certain "linking form" on the torsion subgroup of H2.

If you drop the simply-connected assumption, things get rather harder--indeed if n>3 then any finitely presented group is the fundamental group of a closed n-manifold (which can be constructed in a canonical way given a presentation), and Markov (son of the probabilist) showed that the impossibility of algorithmically distinguishing whether two presentations yield the same group translates to the impossibility of algorithmically classifying manifolds. Even assuming you already knew the fundamental groups were isomorphic, there are still complications beyond what happens in the simply-connected case, but these can sometimes be overcome with the s-cobordism theorem.

In a somewhat different direction, in dimension 3 one can represent manifolds by link diagrams, and Kirby showed that two such manifolds are diffeomorphic (which in dimension 3 is equivalent to homeomorphic) iff you can get from one diagram to the other by a sequence of moves of a certain kind. (see Kirby calculus in Wikipedia; similar statements exist in dimension 4). I suppose that one could argue that this isn't an example of what you were looking for, since if one felt like it one could extract diffeomorphisms from the moves in a fairly explicit way, and one can't (AFAIK) just directly extract some invariants from the diagrams which completely determine whether the moves exist.