Can you give me a reference book where homotopy groups of O(n) are calculated?
In practice, this isn't really possible. There are fibre sequences $$O(n1) \to O(n) \to S^{n1}$$ which allow you to inductively compute the homotopy groups of $O(n)$ in terms of the homotopy of $S^{k}$, for $k < n$. But the latter is one of the main open questions in homotopy theory. Of course, real Bott periodicity tells you the homotopy groups of $O = \lim_{n\to \infty} O(n)$. By the previous fibre sequence, this is the same as $\pi_k(O(n))$ for $n>k+1$  the homotopy groups stabilise at that point  since $\pi_k(S^{n1}) = 0$ in that range. But the higher homotopy of $O(n)$ for a fixed $n$ is less tractable. A good reference for what you can do with the fibre sequence above (and others like it) is MimuraToda's Topology of Lie Groups, I and II. 

