Someone mentioned that the category $\mathbf{Man}$ of (topological) manifolds is really badly behaved, but $\mathbf{Top}$ (the category of topological spaces) isn't much better eithter. It's both complete and cocomplete, but it is not cartesian closed (and thus not a topos).

There are two solutions to this problem: either you generalize what is meant by a space, or you restrict your attention to a collection of "nice" spaces. nLab has a list of "nice" categories of "spaces" which behave better than $\mathbf{Top}$.