How to define homotopy groups in categories as in Quillen's definition for Higher algebraic Ktheory: K_i(M)=\pi_{i+1}(BQM, 0), where M is a small category and BQM is the classifying space of QM. thank you.

In this definition BQM can be taken to be a space  the geometric realization of the nerve of the category QM. The homotopy groups are then the usual homotopy groups from topology. There also is a definition of the homotopy group of a simplicial set  you can thus compute the homotopy group of the nerve without passing to the geometric realization first  and the definition you give looks more like that, but the answers are isomorphic. 


Quillen shows at the beginning of his article on higher algebraic Ktheory that you can calculate the fundamental group $\pi_1(C,a)$ of a category $C$ at an object $a$ by forming the localisation $C[Mor(C)^{1}]$ at all arrows, then by taking $Hom(a,a) = Aut(a)$ in this groupoid. There are size issues, clearly, but for essentially small $C$ these can be ignored. 

