I'm coming a bit late to this party, but I'll put in my two cents anyway because they are rather different from everything else I've heard so far. In a nutshell, my response is:
Yes, I agree that this is a problem (though I do think you would have done better to post only the question and not the rant), and
What you can do is be part of the solution.
For a long time I resisted "homotopical higher category theory" too, for reasons that I think are not unrelated to yours. I even wrote a somewhat whiny blog post about it. What eventually "brought me on board" was not the applications to algebraic geometry or what-have-you (which is not, of course, to denigrate those applications), but the truly category-theoretic conceptual insights arising from what you call HTT. Examples include:
Colimits in a 1-category cannot be as well-behaved as we would like them to be, and the reason is because a 1-category doesn't have enough "room"; an $(\infty,1)$-category fixes this. For instance, Giraud's axioms for a 1-topos assert "descent" only for coproducts and quotients of equivalence relations; the analogous axioms for an $(\infty,1)$-topos assert descent for all colimits.
Passing "all the way to $\infty$" has a "stabilizing" effect that enables $(\infty,1)$-categories and $(\infty,1)$-category theory to "describe itself" in ways that 1-category theory can only approximate. For instance, the 1-category of 1-categories does not include enough information to characterize the "correct" notion of "sameness" for 1-categories, namely equivalence (at least, not unless you hack it with something like a Quillen model structure); for that you need the 2-category of 1-categories, or at least the $(2,1)$-category of 1-categories. But the $(\infty,1)$-category of $(\infty,1)$-categories does characterize them up to the correct notion of equivalence. (Although for many purposes one still needs the $(\infty,2)$-category of $(\infty,1)$-categories, pointing towards the still largely-unexplored territory of $(\infty,\infty)$-categories.) Similarly, a 1-topos can only have a subobject classifier, classifying those objects that are "internally $(-1)$-categories", i.e. truth values; but an $(\infty,1)$-topos can have an object classifier that classifies all objects (up to size limitations).
Various mysterious phenomena in 1-topos theory are explained as shadows of $(\infty,1)$-topos-theoretic phenomena. For instance, the analogy between open geometric morphisms and locally connected ones is explained by seeing them as the steps $k=-1$ and $k=0$ of a ladder of locally $k$-connected $(\infty,1)$-geometric morphisms, and similarly for proper and tidy geometric morphisms. Moreover, various apparently ad hoc notions of the "homotopy theory of toposes", such as cohomology, fundamental groups, shape theory, and so on, are explained as manifestations of the $(\infty,1)$-topos-theoretic "shape", which is characterized by a simple universal property.
Perhaps most importantly, the fundamental idea that the basic objects of mathematics are not just sets, but $\infty$-groupoids. Thus, for instance, the really good notion of "ring" should be an $\infty$-groupoid with a coherent multiplication and addition structure (i.e. a ring spectrum), including the set-based notion of "ring" as simply a special case. And so on.
Note that none of these ideas depends on any concrete model for $(\infty,1)$-categories, and most of them have nothing to do with homotopy theory; they are purely category-theoretic ideas. So I think even a category theorist who cares nothing about homotopy theory ought to be interested in a kind of "category theory" where these are true.
That said, I think a good category theorist should care at least somewhat about homotopy theory, if for no other reason then for the same reason that a good category theorist should care about other applications of category theory. Like all fields of mathematics, category theory is supported and invigorated by its connections to other fields of mathematics, and the close tie between higher category theory and homotopy theory has great potential to stimulate both subjects. That this potential has been realized more fully on the homotopy-theoretic side is, I think, largely an accident of history and personality.
Why is $(\infty,1)$-category theory not usually done "Australian-style"? I believe it is just because people doing $(\infty,1)$-category theory don't know, or at least don't appreciate, Australian-style 1- and 2-category theory, while many Australian-style category theorists don't know or appreciate $(\infty,1)$-category theory. This creates a tremendous opportunity for anyone who is willing to put in the effort to be a bridge, teaching category theorists how to think about $(\infty,1)$-categories "category-theoretically" and teaching $(\infty,1)$-category theorists the benefits of "really thinking like a category theorist".
One way to be such a bridge is to learn the simplicial technology that's currently used for $(\infty,1)$-category theory and "do them Australian-style". For instance, as far as I know there is still no $(\infty,2)$-monad theory with the power and flexibility of 2-monad theory; someone should do it. Enriched $(\infty,1)$-categories are only starting to be investigated. The $(\infty,2)$-category of $(\infty,1)$-profunctors has been used for some applications, but its category-theoretic potential is largely unexplored. As far as I know, no one has even defined $\infty$-double-categories yet. What about generalized $\infty$-multicategories? Etc. etc.
While a worthy endeavor, I suspect that this is not what you want to do. In particular, it sounds like you don't feel able to spend the time to really understand simplicial technology. I can sympathize with that; it's difficult enough for me, and I was already exposed to lots of simplicial stuff as a graduate student since my advisor was an algebraic topologist. So I generally avoid using simplicial technology as much as possible. One way to do this, which I have pursued myself, is to study $(\infty,1)$-categories using 1- and 2-categorical machinery, including Quillen model categories (which, by the way, have an algebraic version that is rather more pleasing to a category theorist's heart) but also homotopy-level structures such as derivators, homotopy 2-categories, and homotopy proarrow equipments.
This works quite well for surprisingly many things, and doesn't require you to learn any simplicial technology. However, it does often depend on the fact that someone has proven something using simplicial technology in order to "get into the world" where you're working. Moreover, you've also expressed some skepticism about the very idea of simplicial technology and concrete models. I think it'd be good if you can get over this to a degree — mathematics has to move forward with what we have, even if it's not perfect, and later on someone can make it better — but I do also sympathize with it, because for instance of the last conceptual insight I mentioned above:
- The basic objects of mathematics are not just sets, but $\infty$-groupoids.
How can this be, if an $\infty$-groupoid is defined in terms of sets (e.g. as a Kan complex)?
Well... there is now a way to study $\infty$-groupoids directly, without defining them in terms of sets: it's called homotopy type theory (HoTT). HoTT is (among other things) a foundational theory, on roughly the same ontological level as ZFC, whose basic objects can be regarded as $\infty$-groupoids; I wrote a philosophical introduction to it from this perspective. (There's also work on an analogous theory whose basic objects are $(\infty,1)$-categories.) Thus, HoTT offers the promise of an approach to homotopy theory and higher category theory that's almost completely free of simplicial technology, and incorporates the conceptual insights of $(\infty,1)$-category theory "from the ground up", allowing us to build intuition for, and work directly with, higher-categorical and higher-homotopical structures without having to construct them explicitly out of sets. When I read or write a proof in $(\infty,1)$-topos-theoretic language, I'm never quite sure whether I've dotted enough "i"s to make all the coherence come out right; but when I instead write it in HoTT then I am, not only because with HoTT I understand the profound reason why you already know what things intimately are (as you put it), but because a HoTT proof can be formalized and verified with a computer proof assistant. There are already some graduate students who have "grown up" with HoTT and can "think in it" in ways that surpass those of us who "came to it late".
Now, this "promise" of HoTT is not yet fully realized. Many coherent higher-categorical structures can be represented simply and conceptually in HoTT; but many others we don't know how to deal with yet. So here's another way you can be part of the solution: improve the ability of HoTT to represent higher category theory, so that eventually it becomes powerful enough that even the "applied" $(\infty,1)$-category theorists can do away with simplices. This is, in large part, what I am now working on myself.
