Definitions
I think there is a definition that should fit into most models of $(\infty,1)$-categories. If you want an "elevator speech" answer, it's:
Definition. A limit of a diagram $\mathcal D \to \mathcal C$ is a terminal object in the $(\infty,1)$-category of objects living over $\mathcal D$.
This is the (almost naive) generalization of one definition for limits in usual category theory. But let me elaborate on this definition.
(Full disclosure, I almost always work with quasicategories, so I anticipate that a better answer can be given by someone who's worked with all models. I hope this answer will be helpful regardless. Also, when you say "limit," I assume you mean homotopy limit. I sometimes distinguish between the two since not everybody is happy when I use classical terminology with an implicit "$\infty$" or "homotopy" before every word.)
Morally speaking, a good model for "$(\infty,1)$-categories" should have definitions for the following ideas:
- Mapping spaces. That is, given an $(\infty,1)$-category
$\mathcal C$, between any two
objects $X,Y$ of $\mathcal C$, a
topological space of morphisms
${\mathcal C}(X,Y)$. This is a
fairly obvious pre-requisite because $(\infty,1)$-categories are supposed to be like categories enriched in spaces.
- Terminal objects. Morally, these are objects $\ast$ such that
for any other object $Y$ in your
$(\infty,1)$-category $\mathcal C$,
the mapping space ${\mathcal
C}(Y,\ast)$ is contractible. There may be more subtle issues involved in defining terminal objects properly, depending on your model, but at least in the case of quasi-categories, it turns out this moral definition is perfectly fine as an actual one. (See Corollary 1.2.12.5 of HTT.)
- Under/Over-Categories, aka Cone Categories. Given two $(\infty,1)$-categories $\mathcal C$ and $\mathcal D$, an $(\infty,1)$-category of ( $(\infty,1)$-) functors between them. And in our discussion, we specifically want the following: Given a diagram ${\mathcal D} \to \mathcal C$, a good notion of an ( $(\infty,1)$-)category whose objects are functors from $\ast \star {\mathcal D}$ to $\mathcal C$, where $\ast \star {\mathcal D}$ is the category obtained by affixing an initial object to $\mathcal D$. This is the same thing as the category of objects of $\mathcal C$ equipped with a map to the diagram $\mathcal D \to \mathcal C$.
You can see why this third point, about cone categories, is so simple in the quasi-category model. It is as simple as defining the join of simplicial sets, and knowing what the mapping space is between simplicial sets.
Anyhow, if you believe that your model (whatever it is) has definitions for the above three things, you can define a limit to be a terminal object in a cone category. You can dually define colimits as initial objects in an undercategory.
Actually proving that (co)limits are preserved.
I assume you wanted an answer that was more specific about actually computing (homotopy) limits using different models (complete Segal spaces, quasi-categories, Kan simplicial categories, et cetera) but I'm afraid I don't know much about comparing homotopy limit computations in different models. Lurie does, however, prove in HTT (Theorem 4.2.4.1) that the usual homotopy (co)limits you'd compute in a category enriched over Kan complexes will agree with the homotopy (co)limits you'd compute in the quasi-category model. So that's a good start! And if you believe in the equivalences between different models of $(\infty,1)$-categories (see for instance Julie Bergner's "A Survey of $(\infty,1)$-Categories") then the equivalences should preserve initial objects of cone categories, so this would be an argument that all models preserve (homotopy) (co)limits.
Why "everybody" takes simplicial sets.
Actually, a lot of people prefer to use other models like the Segal space model. But you can see that with the combinatorics of quasi-categories, a lot of things can be defined and proved fairly cleanly, as I pointed out in some of my commentary above. So that's one advantage of Joyal's quasi-category model. But there are many situations in which the space of objects is so naturally a space that you might prefer a model which isn't based on weak Kan complexes. For instance, in Galatius-Madsen-Tillman-Weiss, they think of the category of cobordisms as a category with a space of objects and a space of morphisms. This model might make it easier, for instance, to compute the classifying space of an $(\infty,1)$-category. And if you were interested in computing a (co)limit of a functor mapping such an $(\infty,1)$-category into another, you wouldn't want to say that your diagram comes from a simplicial set.
Simplicial Sets as the Diagram
Also, it seems you're interested in why Lurie takes as the diagram a map $\mathcal{D} \to \mathcal C$ in which $\mathcal D$ is a simplicial set. I don't think I would take "simplicial set" as the important idea here; I think it's more that $\mathcal D$ is an $(\infty,1)$-category, so for Lurie, it's a weak Kan complex, and in particular a simplicial set. I'm not sure (as Moosbrugger alludes to above) why the generality of an arbitrary simplicial set is so important, but in general I think we would take a diagram $\mathcal D \to \mathcal C$ to be a map of $(\infty,1)$-categories. That is, $\mathcal D$ being a simplicial set isn't so important for this definition. It just needs to be an $(\infty,1)$-category in whatever model you're using, and in the Lurie example, you should probably think of it as a quasi-category, rather than just an arbitrary simplicial set. And you can always replace an arbitrary simplicial set with a weak Kan complex (these are the fibrant-cofibrant objects in the Joyal model category.)