I am interested in clean algorithms for approximating solutions and so I am interested in numerical analysis, but most of the books I have seen get bogged down in error analysis or they spend a lot of time and effort in squeezing an additional 2% efficiency from a classical algorithm. What are some works which just have the fundamental ideas and methods?

Are you looking for a reference that links the field of numerical analysis to mathematical concepts moreso than algorithmic concepts? Matrix Computations by Golub and Van Loan is a fairly important book that studies the algebraic structures of matrices and derives algorithms from those properties. If you're looking for an entrylevel work, I keep a copy of Michael Heath's book Scientific Computing on my desk. It covers fundamental concepts and algorithms fairly well, in my opinion. Do you have a specific problem domain in mind? 


To augment Timur's answer:
I've picked these books for their balance of important algorithms and key insights, delivered with clear prose. I also like Strikwerda's book on finite difference methods, and the HairerWanner books on numerical methods for ODE. But these focus a lot on error analysis, which may not be what you wish. 


One of my favorites is "Numerical Methods that Work" by Forman S. Acton. It's an old book; some of the examples seem quaint now that we have far more compute power. But the principles haven't changed. Acton's book is good about illustrating technique, learning how to think like a numerical analyst. 


My favorites:


