In the topological category the usual compact-open topology does the job. (EDIT: Or rather I suppose you might have to modify it a little so that $h\mapsto h^{-1}$ is continuous.) At times you might want to replace a function space by the simplicial set called its total singular complex. This is an instance of the more general procedure of replacing a space by its total singular complex when convenient, a trend that began around that same time. But there may not be any great technical advantages in doing so when working with spaces of homeomorphisms.
In the $PL$ category it gets worse: I don't know a useful topology on the group of $PL$ automorphisms of $M$. Topologizing it as a subspace of the space of homeomorphisms can't be right, and there isn't something like a Whitney topology to fix things up.
EDIT: This seems about right as a reason why $PL$ homeomorphisms cannot be treated as a subspace of homeomorphisms. Suppose $h$ is a $PL$ automorphism of $\mathbb R^n$ fixing the origin. In a neighborhood of the origin it is radial (linear on each ray). By a sort of reverse Alexander trick you would be able to make it radial everywhere if you were allowed to use isotopies through $PL$ homeomorphisms rather than actual $PL$ isotopies. In this way you would get a deformation retraction to the subgroup consisting of radial $PL$ homeomorphisms. Then you could further deform to the smaller group that consists of those radial homeomorphisms that are made by coning $PL$ homeomorphisms of a ($PL$) $(n-1)$-sphere. That would mean that the canonical map from $Aut^{PL}(D^n)\simeq Aut^{PL}(S^{n-1})$ to $Aut^{PL}(D^n-S^{n-1})$, or to germs at the origin, is an equivalence. But the fiber of this map to germs is the pseudoisotopy space $P^{PL}(S^{n-1})$, and this is not contractible. Contradiction. Why is $P^{PL}(S^{n-1})$ not contractible? It fibers over the pseudoisotopy embedding space $PE^{PL}(D^{n-1},S^{n-1})$, with contractible fiber $P^{PL}(D^{n-1})$. The space $PE^{PL}(D^{n-1},S^{n-1})$ is not contractible because (1) the smooth analogue $PE^{Diff}(D^{n-1},S^{n-1})$ is contractible by Hatcher's light bulb trick, (2) the fiber of $PE^{Diff}(D^{n-1},S^{n-1})\to PE^{PL}(D^{n-1},S^{n-1})$ is equivalent by smoothing theory to $\Omega fiber (O_n/O_{n-1}\to PL_n/PL_{n-1})$, and (3) this last is not contractible. Why (3)? Because if it were contractible then the fiber of the map $P^{Diff}(D^{n-1})\to P^{PL}(D^{n-1})$, which is $\Omega^n fiber (O_n/O_{n-1}\to PL_n/PL_{n-1})$ by smoothing theory, would be contractible. But $P^{PL}(D^{n-1})$ is contractible while $P^{Diff}(D^{n-1})$ is not by Waldhausen (rationally it becomes $\Omega^2 K(\mathbb Z)$ as $n\to \infty$).