Disclaimer: I won't be extremely precise about the set theoretic foundations in the question as I feel these are not the most important points in it (I apologize in advance for any set theoretic mistakes or inaccuracies in the question - my set theory background is not very extensive). In particular I will assume there are enough large cardinals to make everything that follows work.
It could help to imagine for the moment we are watching a sort of game. In this game there are 2 players. The Theorist and the Skeptic. The Theorist and the Skeptic both agree on the $2$-category theory of small and large $1$-categories. That is the fully faithfully nested $2$-categories (the objects in the last one form a class or an even larger cardinal - I won't stress this point as I believe it won't be extremely important in what follows).
$$\mathsf{Cat}^{}_1 \subset \mathsf{CAT}_1$$
The Theorist then puts forth his favorite model of $\infty$-categories. Alas the Skeptic being as such, is very much inclined to not accept the model proposed by the Theorist. More troublesome is his attitude though.
The Skeptic does not accept any other model of $\infty$-category theory out there.
The Skeptic will only accept arguments made in the standard set theoretic foundations of mathematics, being lenient with large cardinal axioms when necessary (whatever that means). In particular he will not accept arguments made in alternate foundations like HoTT etc.
The Skeptic is willing to accept only $1$-categorical and $2$-categorical concepts. He will not accept model categories as more than than a tool to prove stuff about the their homotopy categories etc...
The Skeptic refuses to look at the internal structure of the model. He will only accept "external" properties as evidence for the validity of the model.
Keeping the above restrictions i mind the Skeptic interrogate the Theorist.
Sizes - The Skeptic asks the Theorist to consult his model and provide him with fully faithfully nested homotopy $2$-categories $$\mathsf{Cat}_{\infty} \subset \mathsf{CAT}_{\infty}$$ Whose objects are respectively small and large $(\infty,1)$-categories. With functors as 1-morphisms and equivalence classes of natural transformations as $2$-morphisms.
Compatibility with classical category theory - The Theorist must provide fully faithful right adjoint inclusions, $\mathsf{Cat}_1 \subset \mathsf{Cat}_{\infty}$ and $\mathsf{CAT}_1 \subset \mathsf{CAT}_{\infty}$, compatibly with (1). We will denote their left adjoints by $ho(-)$.
Cartesian closed - Skeptic verifies that all the $2$-categories in (1) are cartesian closed (this gives us functor $\infty$-categories, in particular diagram categories).
Subategory of $\infty$-groupoids - Using the $1$ -category $\Delta^1$ we get from $(2)$ and diagram categories we get from $(3)$ he can define a full subcategory of small $\infty$-groupoids $\mathsf{Grpd}_{\infty} \subset \mathsf{Cat}_{\infty}$ on those $\infty$-categories all of whose arrows are invertible.
The $\infty$-category of spaces - The Skeptic defines a pseudo-functor $LFib_{ /-} : \mathsf{CAT}_{\infty}^{op} \to \mathsf{CAT}_{1}$ by the following: $$\mathcal{C} \mapsto LFib_{/ \mathcal{C}} := \{ \text{functors } \mathcal{E} \to \mathcal{C} \text{ with small fibers and s.t. the fibers of }$$ $$ \mathcal{E}^{\Delta^1} \to \mathcal{E}^{\{0\}} \times_{\mathcal{C}^{\{ 0\}}} \mathcal{\mathcal{C}^{\Delta^1}} \text{ are all terminal} \}$$ He proceeds to verify that it is represented via the 2-yoneda embedding by an $\infty$-category $\mathcal{S} \in \mathsf{CAT}_{\infty}$, the $\infty$-category of spaces.
Homotopy Hypothesis - Denoting by $Top$ the ordinary $1$-category of (small) topological spaces and $W$ the (weak) homotopy equivalences. The Skeptic defines an $\infty$-catgory $Top[W^{-1}] \in \mathsf{CAT}_{\infty}$ by the universal property that for every $\infty$-category $\mathcal{C} \in \mathsf{CAT}_{\infty}$ the restriction $\mathcal{C}^{Top[W^{-1}]} \to \mathcal{C}^{Top}$ exhibits the domain as the full subcategory on the functors which send $W$ to equivalences in $\mathcal{C}$ (if such a category doesn't exist in the Theorist's model he loses of course). The Theorist must supply an equivalence of $\infty$-categories $Top[W^{-1}] \cong \mathcal{S}$.
The internal $\infty$-category of $\infty$-categories - The Skeptic repeats step $(5)$ only this time with the pseudo-functor $CoCart_{/-} : \mathsf{CAT}_{\infty}^{op} \to \mathsf{CAT}_1$ of (small) cocartesian fibrations as well as $CoCart^{\le 1}_{/ -}$ of cocartesian fibrations whose fibers are $1$-categories. He then verifies that both are representable and the natural inclusion $CoCart^{\le 1}_{/-} \hookrightarrow CoCart_{/-}$ is represented by a fully faithfull right adjoint $\mathfrak{Cat}_1 \hookrightarrow \mathfrak{Cat}_{\infty} \in \mathsf{CAT}_{\infty}$. _
Remark: I am assuming in $(7)$ that the $1$-category whose objects are cocartesian fibrations over a fixed $\infty$-category and whose morphismsare equivalence classes of functors can be constructed using only the homotopy $2$-truncation of the full $(\infty,2)$-category of $\infty$-categories. I am not entirely sure whether this is a valid assumption. I think the work of Emily and Verity at least suggests that some version of this might be true. Question: Is this step really possible?
Compatibility with the internal category of $1$-Categories - The Theorist must supply an equivalence $ho(\mathfrak{Cat}_1) \cong \mathsf{Cat}_1 \in \mathsf{CAT}_1$.
Compatibility with $\mathcal{S}$ - The skeptic verifies that the obvious natural transformation between the pseudo-functors from $(5)$ and $(7)$ gives rise to a fully faithful inclusion $\mathcal{S} \hookrightarrow \mathfrak{Cat}_{\infty}$ admitting both a left and right adjoint denoted by $|-|$ and $(-)^{\cong}$ respectively.
(Co-)Completness - The Skeptic verifies that $\mathfrak{Cat}_{\infty}$ admits all (small) $\infty$-limits/colimits. Meaning that for all small $\infty$-categories $\mathfrak{I} \in \mathsf{Cat}_{\infty}$ the constant diagram functor $\Delta : \mathfrak{Cat}_{\infty} \to \mathfrak{Cat}_{\infty}^{\mathfrak{I}}$ admits both left and right adjoint (it then follows from $(9)$ that the same holds for $\mathcal{S}$).
Internal cartesian closeness - The Skeptic verifies that $\mathfrak{Cat}_{\infty}$ is cartesian closed (i.e. that the right adjoint to $\Delta : \mathfrak{Cat}_{\infty} \to \mathfrak{Cat}_{\infty}^{\times 2} \in \mathsf{CAT}_{\infty}$ from $(10)$ admits a further right adjoint).
Compactness of $\bullet$ and $\Delta^1$ - By $(8)$ can think of small $1$-categories as objects in $\mathfrak{Cat}_{\infty}$ in particular we have the objects $\bullet, \Delta^1 \in \mathfrak{Cat}_{\infty}$. The Skeptic verifies that $(-)^{\cong} : \mathfrak{Cat}_{\infty} \to \mathcal{S}$ and $((-)^{\Delta^1})^{\cong} : \mathfrak{Cat}_{\infty} \to \mathcal{S}$ preserve $\infty$-colimits indexed by filtered $1$-categories.
Compact Generation - The Skeptic verifies that any full subcategory of $\mathfrak{Cat}_{\infty}$ closed under all (small) $\infty$-limits/colimits containing $\bullet, \Delta^1$ is already $\mathfrak{Cat}_{\infty}$. Similarly for $\mathcal{S}$ and $\bullet \in \mathcal{S}$.
Free Generation of $\mathcal{S}$ - For any cocomplete (large) $\infty$-category $\mathcal{C} \in \mathsf{CAT}_{\infty}$. The Skeptic defines the full subcategory $Fun^{cont}(\mathcal{S},\mathcal{C}) \subset \mathcal{C}^{\mathcal{S}}$ of the functor category (which exists by cartesian closeness) whose objects are the functors which preserve all (small) $\infty$-colimits. He verifies that evaluation induces an equivalence $Fun^{cont}(\mathcal{S}, \mathcal{C}) \cong \mathcal{C} \in \mathsf{CAT}_{\infty}$.
Self Consistency - Using $(11)$, $(7)$, $(8)$ and $(2)$ the Skeptic defines a natural enrichment of $ho(\mathfrak{Cat}_{\infty}) \in \mathsf{CAT}_1$ over $ho(\mathfrak{Cat}_1) \cong \mathsf{Cat}_1$ to obtain the homotopy $2$-category $ho_2(\mathfrak{Cat}_{\infty})$ of the internal $\infty$-category of small $\infty$-categories. The Theorist must supply an equivalence of $2$-categories $ho_2(\mathfrak{Cat}_{\infty}) \cong \mathsf{Cat}_{\infty}$ (compatible with (8)).
Compatibility of spaces & groupoids - The Skeptic uses the equivalence from $(15)$ to identify the subcategory $\mathsf{Grpd}_{\infty} \subset \mathsf{Cat}_{\infty}$ with a full subcategory $\mathsf{Grpd}_{\infty} \subset \mathfrak{Cat}_{\infty}$. He then verifies that it is essentially equal to $\mathcal{S}$ (i.e. they are both full subcategories on the same equivalence classes of objects).
I am sure several of the above steps are redundant as they follow from a combination of some of the other steps. I realize that this might be an incredibly tall order to ask for but I suddenly feel bold enough to ask this (foolishly no doubt...).
Question (imprecise): Could these requirements be strong enough to fix the homotopy $2$-category theory of $\infty$-categories completely?
To make the question precise one can imagine instead of just small and large the Theorist has to supply the homotopy $2$-category $CAT^{\kappa}_{\infty}$ of $\infty$-categories of size $\kappa$ for every strongly inaccessible cardinal $\kappa$ together with compatible inclusions (probably you want to restrict to some nice family of cardinals which are consistent with each other, not entirely sure about this set theoretic issue to be honest...). Then do all the steps above for very inclusion of cardinals $\kappa_1 \in \kappa_2$ with "small" replaced by $\kappa_1$ and "large" replaced by $\kappa_2$. Then the question becomes:
Question (quasi-precise): Are any two collections of $2$-categories $\{CAT^{\kappa}_{\infty}\}_{\kappa}$ satisfying the list of requirements above actually equivalent? (as a filtered diagram of 2-categories compatible with all the additional data provided by the Theorist along the way in the list above).