I have two comments to make.
First, I suggest taking a second look at Milnors book. I found his treatment of chern classes in one of the appendices really clear and enlightening. I don't have acces to his book at the moment and think he doesn't really state the Chern Weil theorem, but that should only be a small step away. You could also take a look at chapters 5.3 to 5.5 in Moritas "Geometry of Differential Forms". I think page 196 in particular might be enlightening. Chapter 6 is devoted to Chern-Weil theory for principal bundles.
Second, I don't think one can really reduce to the case of the trivial connection because vector bundles are locally trivial because trivialisations in general are not connection preserving. There is a connection preserving trivialisation to the flat connection on the trivial bundle if and only if the connection is locally flat there.