I've been told that it's important to know modern physics, Differential Geometry and Algebraic Topology for understanding higher structures. Is there any other prerequisite for understanding Lurie's work? Since the title of the book indicates, I guess Algebraic Geometry is also important. Please tell me if I'm wrong. Moreover how deep should I known on those subjects and others I do not mention?
To read Higher Topos Theory, you'll need familiarity with ordinary category theory and with the homotopy theory of simplicial sets (Peter May's book "Simplicial Objects in Algebraic Topology" is a good place to learn the latter). Other topics (such as classical topos theory) will be helpful for motivation.
To read "Higher Algebra", you'll need the above and familiarity with parts of "Higher Topos Theory". Several other topics (stable homotopy theory, the theory of operads) will be helpful for motivation.
To read the papers "Derived Algebraic Geometry ???", you need all of the above plus familiarity with Grothendieck's theory of schemes, along with some more recent ideas in algebraic geometry (stacks, etcetera).
Since no knowledge of modern physics was required to write any of these books and papers, I can't imagine that you need any such knowledge to read them.