I don't think I have a compelling answer to this question, but maybe some bits and pieces that will be helpful. One point is that all of the examples that you bring up are related to the first: simplicial sets can be used as a model for the homotopy theory of spaces. Pretty much any homotopy theory can be "described" in terms of the homotopy theory of spaces, just like any category can be "described" in terms of the category of sets (via the Yoneda embedding, for example). So if you've decided that "space" means simplicial set, then it's pretty natural to start thinking about presheaves of simplicial sets when you want to think about the homotopy theory of (pre)sheaves of spaces, as in motivic homotopy theory.
But that just brings us to the question "why use simplicial sets as a model for the homotopy theory of spaces"? It's certainly not the only model, and some alternatives have been listed in the other responses. Another alternative is more classical: the category of topological spaces can be used as a model for the homotopy theory of spaces. So, you might ask, why not develop the theory of the cotangent complex using topological commutative rings instead of simplicial commutative rings? There's no reason one couldn't do this; it's just less convenient than the alternative.
There are several things that make simplicial sets very convenient to work with.
The category of simplicial sets is very simple: it is described by presheaves on a category with not too many objects and not too many morphisms, so the data of a simplicial set is reasonably concrete and combinatorial. The category of topological spaces (say) is more complicated in comparison, due in part to pathologies in point-set topology which aren't really relevant to the study of homotopy theory.
The category of simplices is (op)-sifted. This is related to the concrete observation that the formation of geometric realizations of simplicial sets (or simplicial spaces) commutes with finite products. More generally it guarantees a nice connection between the homotopy theory of simplicial sets and the homotopy theory of bisimplicial sets, which is frequently very useful.
The Dold-Kan correspondence tells you that studying simplicial objects in an abelian category is equivalent to studying chain complexes in that abelian category (satisfying certain boundedness conditions). So if you're already convinced that chain complexes are a good way to do homological algebra, it's a short leap to deciding that simplicial objects are a good way to do homological algebra in nonabelian settings. This also tells you that when you "abelianize" a simplicial construction, you're going to get a chain complex (as in the story of the cotangent complex: Kahler differentials applied to a simplicial commutative ring yields a chain complex of abelian groups).
Simplicial objects arise very naturally in many situations. For example, if U is a comonad on a category C (arising, say, from a pair of adjoint functors), then applying iterates of U to an object of C gives a simplicial object of C. This sort of thing comes up often when you want to study resolutions. For example, let C be the category of abelian groups, and let U be the comonad U(G) = free group generated by the elements of G (associated to the adjunction {Groups} <-> {Sets} given by the forgetful functor,free functor). Then the simplicial object I just mentioned is the canonical resolution of any group by free groups. Since "resolutions" play an important role in homotopy theory, it's convenient to work with a model that plays nicely with the combinatorics of the category of simplices. (For example, if we apply the above procedure to a simplicial group, we would get a resolution which was a bisimplicial free group. We can then obtain a simplicial free group by passing to the diagonal (which is a reasonable thing to do by virtue of (2) )).
Simplicial sets are related to category theory: the nerve construction gives a fully faithful embedding from the category of small categories to the category of simplicial sets. Suppose you're interested in higher category theory, and you adopt the position that "space" = "higher-groupoid" = "higher category in which all morphisms are invertible". If you decide that you're going to model this notion of "space" via Kan complexes, then working with arbitrary simplicial sets gives you a setting where categories (via their nerves) and higher groupoids (as Kan complexes) both sit naturally. This observation is the starting point for the theory of quasi-categories.
All these arguments really say are that simplicial objects are nice/convenient things to work with. They don't really prove that there couldn't be something nicer or more convenient. For this question I'd just offer a sociological argument. The definition of a simplicial set is pretty simple (see (1)), and if there was a simpler definition that worked as well, I suspect that we would be using it already.