Question

Can you give me a reference book where homotopy groups of O(n) are calculated?

Answer 1

In practice, this isn't really possible. There are fibre sequences

$$O(n-1) \to O(n) \to S^{n-1}$$

which allow you to inductively compute the homotopy groups of $O(n)$ in terms of the homotopy of $S^{k}$, for $k < n$. But the latter is one of the main open questions in homotopy theory.

Of course, real Bott periodicity tells you the homotopy groups of $O = \lim_{n\to \infty} O(n)$. By the previous fibre sequence, this is the same as $\pi_k(O(n))$ for $n>k+1$ -- the homotopy groups stabilise at that point -- since $\pi_k(S^{n-1}) = 0$ in that range. But the higher homotopy of $O(n)$ for a fixed $n$ is less tractable.

A good reference for what you can do with the fibre sequence above (and others like it) is Mimura-Toda's Topology of Lie Groups, I and II.