You might want to look at Section 3.1 of "Group Theory and Physics" by Shlomo Sternberg, Cambridge Univ Press, 1994. This explains, through a simple example, how (in Sternberg's words) "molecular spectroscopy is an application of Schur's lemma". The argument is elementary in nature. The last chapter of the book by James & Liebeck (Representations and Characters of Groups 2e, Cambridge Univ Press, 2001) is a longer exposition of the same idea. I notice another post here about work by Diaconis - Diaconis has a book called "Group Representations in Probability and Statistics" which is available for free download. See the link at
This page has links to dozens of useful articles. Also, there is the book "Unitary Group Representations in Physics, Probability and Number theory" by George W Mackey (Benjamin/Cummings Publ Co, 1978). This is more advanced than the others though. For applications to quantum chemistry there is (amongst many) "Chemical Applications of Group Theory" by F Albert Cotton, published by John Wiley. If you want to see how Section 2.7 of Serre's book is actually used in practice by chemists, see Chapter 6 in Cotton's book.