I had great trouble finding a single book that would be good for everything. There is a book Lectures on Differential Geometry by Chern, Chen, and Lam that's pretty nice (although Chern's name on the cover might be affecting my judgment). It has the advantage of being very concise and rather clear.

(Also, I second the suggestion of Milnor's Morse Theory. The appendix to Milnor's Characteristic classes has a very nice exposition of connections, but it has no other Riemannian geometry).

I had great trouble finding a single book that would be good for everything. There is a book Lectures on Differential Geometry by Chern, Chen, and Lam that's pretty nice (although Chern's name on the cover might be affecting my judgment). It has the advantage of being very concise and rather clear.

EDIT: The question asked specifically for Riemannian geometry rather than differential geometry. If I were to describe the above book, I'd say it's mostly about the former, regardless of the title (although it has a few chapters on other topics at the end). However, I'm not sure I understand the difference well enough to judge. It certainly has a chapter on "Riemannian geometry".

(Also, I second the suggestion of Milnor's Morse Theory. The appendix to Milnor's Characteristic classes has a very nice exposition of connections, but it has no other Riemannian geometry).