Let $Vect_{\mathbb{R}}$ be the category of (say, finite dimensional) vector spaces over $\mathbb{R}$. The automorphism group of the object $\mathbb{R}^n\in Vect_{\mathbb{R}}$, is $GL_n(\mathbb{R})$. We usually like to think of it as a topological group. For example $BGL_n(\mathbb{R})$ classifies real $n$-dimensional vector bundles. This suggests that perhaps we can make $Vect_{\mathbb{R}}$ into a topological category in such a way that the topological group $BGL_n(\mathbb{R})$ will be the space of automorphisms of $\mathbb{R}^n$. Namely, we will get an $\infty$-category such that the $BGL_n(\mathbb{R})$-s are the connected components of its space of objects.
The obvious idea is just to take the hom-sets of linear maps with their natural topology, but note that this makes all the mapping spaces contractible. Of course one can make an artificial definition, like taking only isomorphisms or chopping the mapping space into connected components according to the rank, but this has other disadvantages. For example, if we let $Vect_{\mathbb{C}}$ be the category of (finite dimensional) vector spaces over $\mathbb{C}$, we have the extension/restriction of scalars adjunction $$ Vect_{\mathbb{R}}\leftrightarrows Vect_{\mathbb{C}} $$ and we would like to make it into an $\infty$-adjunction. The isomorphism of hom-sets that comes from the adjunction does not respect ranks. So my question is basically,
can one enrich $Vect_{\mathbb{R}}$ and $Vect_{\mathbb{C}}$ over topological spaces in such a way that the restriction/extension of scalars will induce an $\infty$-adjunction and such that the automorphisms of $\mathbb{R}^n$ (resp. $\mathbb{C}^n$) will be equivalent to $GL_n(\mathbb{R})$ (resp. $GL_n(\mathbb{C})$)?