I think the general wisdom is that Deligne's Travaux de Shimura and Milne's Introduction to Shimura Varieties are the most comprehensive references, with the latter being somewhat lighter on prerequisites (but heavier on examples).
I've heard it suggested by people who work in the area that the best way to learn the theory is via special cases and examples, motivated by focused research problems. I suppose this is true of many things, though.
It might also help to learn about quaternionic Shimura curves first, assuming that you know a bit about modular curves.