To compute the homotopy groups of a simplicial set $X$, you need to be able to construct a weak equivalence $X \to Y$ where $Y$ is a Kan complex, and then compute the homotopy groups of $Y$ using the definitions you were discussing.
This might seem circular - you need to detect if $X \to Y$ is an equivalence. However, you can construct $Y$ directly using certain more elementary equivalences. Specifically, for a map of a horn $\Lambda \to X$ we can form the pushout of the diagram $\Delta \leftarrow \Lambda \rightarrow X$, called $X'$; on geometric realizations this is homotopy equivalent because you can construct an explicit retraction. The class of maps $X \to Y$ generated by such pushouts is called the family of anodyne extensions of $X$, and you can always construct an anodyne extension which is a Kan complex by the small object argument (you keep gluing in solutions to every possible horn-filling problem).
If you want a more canonical answer, there is also Kan's $\operatorname{Ex}^\infty$ construction.
If you want another reference, there are Kan's older papers, and my recollection is that Joyal and Tierney has quite a number of details as well.
