This question is inspired by this Tim Gowers blogpost.
I have some familiarity with the work of Jacob Lurie, which contains ideas which are simply astounding; but what I don't understand is which key insight allowed him to begin his programme and achieve things which nobody had been able to achieve before. People had looked at $\infty$-categories for years, and the idea of $(\infty,n)$-categories is not in itself new. What was the key new idea which started "Higher Topos Theory", the proof of the Baez-Dolan cobordism hypothesis, "Derived Algebraic Geometry", etc.?
This question is inspired by this Tim Gowers blogpost.
My answer would be that his insight was firstly that it pays to take what Grothendieck said in his various long manuscripts, extremely seriously and then to devote a very large amount of thought, time and effort. Many of the methods in HTT have been available from the 1980s and the importance of quasi-categories as a way to boost higher dimensional category theory was obvious to Boardman and Vogt even earlier. Lurie then put in an immense amount of work writing down in detail what the insights from that period looked like from a modern viewpoint.It worked as the progress since that time had provided tools ripe for making rapid progress on several linked problems. His work since HTT continues the momentum that he has built up.
As far as `abstracter than thou' goes, I believe that Grothendieck's ideas in Pursuing Stacks were not particularly abstract and Lurie's continuation of that trend is not either. Once you see that there are some good CONCRETE models for $\infty$-categories the geometry involved gets quite concrete as well. Simplicial sets are not particularly abstract things, although they can be a bit scary when you meet them first. Quasi-categories are then a simple-ish generalisation of categories, but where you can use both categorical insights and homotopy insights. That builds a good intuition about infinity categories... now bring in modern algebraic topology with spectra, etc becoming available.
I think, one of the key insights underlying derived algebraic geometry and Lurie's treatment of elliptic cohomology is taking some ideas of Grothendieck really serious. Two manifestations:
1) One of the points of the scheme theory initiated by Grothendieck is the following: if one takes intersection of two varieties just on the point-set level, one loses information. One has to add the possibility of nilpotents (somewhat higher information) to preserve the information of intersection multiplicities and get the "right" notion of a fiber product. Now one of the points of derived algebraic geometry (as explained very lucidly in the introduction to DAG V) is that for homological purposes this is not really the right fiber product - you need to take some kind of homotopy fiber product. This is, because one still loses information because one is taking quotients - one should add isomorphism instead and view it on a categorical level. Thus, you can take a meaningful intersection of a point with itself, for example. This is perhaps an instance where the homological revolution, which went to pure mathematics last century, benefits from a second wave, a homotopical revolution - if I am allowed to overstate this a bit.
2) Another insight of Grothendieck and his school was, how important it is to represent functors in algebraic geometry - regardless of what you want at the end. [as Mazur reports, Hendrik Lenstra was once sure that he did want to solve Diophantine equations and did not want to represent functors - and later he was amused that he represented functors to solve Diophantine equations.] And this is Lurie's approach to elliptic cohomology and tmf: Hopkins and Miller showed the existence of a certain sheaf of $E_\infty$-ring spectra on the moduli stack of elliptic curves. Lurie showed that this represents a derived moduli problem (of oriented derived elliptic curves).
Also his solution of the cobordism hypothesis has a certain flavor of Grothendieck: you have to put things in a quite general framework to see the essence. This philosophy also shines quite clearly through his DAG, I think.
Besides, I do not think, there is a single key insight in Higher Topos Theory besides the feeling that infinity-categories are important and that you can find analogues to most of classical category theory in quasi-categories. Then there are lots of little (but every single one amazing) insights, how this transformation from classical to infinity-category theory works.
One of the things that Jacob Lurie finds important is to "think invariantly": Do not use model dependent definitions. Do not prove model-dependent results.
Here's an example to illustrate what I mean:
"The $\infty$-category of spetra is the free stable $\infty$-category with colimits generated by a single object". That's a nice model independent definition of the $\infty$-category of spectra.
I think one of the insights leading to his successes, which is of course not unique to him, is that when doing higher category theory it is useful to add invertible higher cells going all the way up to the top before you try to add noninvertible ones anywhere. This is along the same lines as Noah's answer: the cobordism hypothesis, which was originally stated in terms of n-categories, becomes easier once you generalize it to a stronger statement about (∞,n)-categories, since then induction becomes easier/possible.
My understanding (which isn't very deep and so someone should please correct me if I'm wrong) is that one key idea in the Hopkins-Lurie proof of the cobordism hypothesis was to generalize the cobordism hypothesis to allow more general types of cobordism categories. This stronger result the becomes easier to prove because they could do a double-induction argument that moves between the (stably n-)framed case (which is the usual cobordism hypothesis) and the oriented case. Furthermore, it means that the final results they get are stronger, because you get a classification not just of framed TFTs, but of TFTs with more general kinds of structures on the bordism categories.
have you tried asking Jacob?