To compute the homotopy groups of a simplicial set, you need to be able to construct a weak equivalence $X \to Y$ where $Y$ is a Kan complex, and then compute the homotopy groups of $Y$ using the definitions you were discussing.

This might seem circular - you need to detect if $X \to Y$ is an equivalence.  However, you can construct $Y$ directly using certain more elementary equivalences.  Specifically, for a map of a horn $\Lambda \to X$ we can form the pushout of the diagram $\Delta \leftarrow \Lambda \rightarrow X$, called $X'$; on geometric realizations this is homotopy equivalent because you can construct an explicit retraction.  The class of maps $X \to Y$ generated by such pushouts is called the family of anodyne extensions of $X$, and you can always construct an anodyne extension which is a Kan complex by the small object argument (you keep gluing in solutions to every possible horn-filling problem).

If you want a more canonical answer, there is also Kan's $\operatorname{Ex}^\infty$ construction.

If you want another reference, there are Kan's older papers, and my recollection is that Joyal and Tierney has quite a number of details as well.