Homotopy function complex for quasi-categories The model structure on sSet for quasi-categories (the Joyal model structure) is not enriched over sSet with the Quillen model structure, so ordinary internal Hom of simplicial sets is not a correct homotopy function complex of quasi-categories. We can give a correct definition using either cosimplicial or simplicial resolutions, which can be defined by the small object argument as usual. But we can give a more explicit description of such resolutions.
For example, we can define a cosimplicial frame on the terminal simplicial set by taking nerves of groupoids $\{ 0 \simeq \ldots \simeq n \}$. Then a cosimplicial frame on a simplicial set can be defined as the cartesian product with this cosimplicial object. This construction can be found, for example, in [1].
The problem with this construction is that it produces too large function complex. For example, a 1-simplex of $\mathrm{hMap}(\Delta^0,X)$ is a 1-simplex of $X$ together with an inverse and higher simplices that guarantee that they are inverses. It seems that there should be a definition of a homotopy function complexes such that 1-simplices in the example above are just 1-simplices of $X$ which are equivalences.
The obvious way to define such homotopy function complex is to take the largest Kan complex contained in the internal Hom. It seems that we also can define a simplicial resolution which produces the same homotopy function complex. Can we construct homotopy function complexes in such a way? Were such constructions described in the literature?
[1] Daniel Dugger and David I. Spivak, MR 2764043 Mapping spaces in quasi-categories, Algebr. Geom. Topol. 11 (2011), no. 1, 263--325.
 A: Yes, you can compute the mapping spaces in ∞-categories by taking the biggest Kan subcomplex of the internal hom.
The trick is not to use the Joyal model structure, but instead the model structure on marked simplicial sets defined in Higher Topos Theory, proposition 3.1.3.7 (in the case $S=\Delta^0$). By proposition 3.1.4.1 the fibrant objects are exactly the quasicategories with the equivalences marked, by corollary 3.1.4.4 it is a simplicial model category and its mapping spaces are defined exactly as you describe and by theorem 3.1.5.1 this is Quillen equivalent to the Joyal model structure in a way that does the obvious thing on fibrant objects.
A: You certainly can take the maximal Kan complex of the quasicategory of maps between quasicategories (note that the mapping quasicategory already has all of the information you want -- you're just forgetting the non-invertible 1-morphisms in order to get at the underlying space).
The standard way of replacing $s\mathcal{S}et_{\mathrm{Joyal}}$ with a Quillen equivalent model category enriched over $s\mathcal{S}et_\mathrm{Quillen}$ (i.e. a simplicial model category) is to use "marked simplicial sets," described e.g. in Riehl's Categorical Homotopy Theory.
A marked simplicial set is a simplicial set along with a distinguished collection of 1-simplices (always including the degenerate simplices) which are regarded as "equivalences". We turn quasicategories into marked simplicial sets by marking the 1-simplices which were already equivalences in the quasicategory.
The reason why marked simplicial sets form a simplicial model category essentially follows from the fact that marked simplicial sets are a subcategory of the category of presheaves on a certain indexing category $\Delta^+$, which looks just like $\Delta$ except we factor $s_0:[1]\to[0]$ through an intermediary object $e$. 1-simplices in the image of $X_e\to X_1$ are understood to be the "equivalences".
Note that in this case, our cosimplicial frame is the standard one in simplicial sets, with each 1-simplex marked.
As you're probably aware, substituting the Joyal model structure for a Quillen equivalent simplicial model category gives us access to a number of technical tools and constructions; for example, we can use weighted limits to calculate homotopy limits as Riehl describes in her book.
I've left a lot of details hazy. If you have further questions, I'll try to clear them up, but you may just want to consult the book I've mentioned, which is available here: http://www.math.jhu.edu/~eriehl/cathtpy.pdf
