First: Learn you some classical electrodynamics! Quantum field theory is after all a theory of fields, so one really should spend some time thinking aboutand classical electrodynamics. It's is the main example of a field theory! Skipping it is a bit like skipping spheres to study homotopy categories.
To your larger question: I think you should not attempt to read the CQFT literature without reading some of the physics literature as well. The CQFT literature doesn't exist in isolation, so studying it alone is at best only hearing part of a conversation. For another, most of the existing textbooks on CQFT are basically research monographs. You're expected to bring to them some intuition for the problem domain. (The exception to this is Dimock's Quantum Mechanics & Quantum Field Theory: A Mathematical Primer, which could only be better if it were longer.)
But the physics literature is very large! And written by people who either enjoy trolling mathematicians or are genuinely unable to distinguish between the $SO(3)$ and $\mathfrak{so}(3)$. So, yes, you have to be choosy. And Since how to be choosy is a matter of taste, so this is where I get opinionated.
too complicated: the details of the calculations obscures the basic structures.
not well understood: e.g., the infrared structure of even trivial theories like the quantum Maxwell theory is one of the major research themes of the past decade, never mind the problem of confinement.
probably broken at a foundational level: It's highly likely that neither quantum electrodynamics nor the Higgs field nor the Standard Model itself actually existexists as a continuum modelsmodel.
Take the other path: Learn QFT from the statistical physicists. It's easier! They have non-trivial solvable examples! In low dimensions! Also: the core organizing principlesprinciple of QFT -- namely renormalization as a means of isolating the low-frequency behavior -- was discovered by people doing statistical physics (Kadanoff & Wilson), so you might as well follow along in their footsteps.
You learned all that differential geometry, so you might as well use it! And thinking about what QFT on curved spacetime should be is a good thing to do, since it forces you to focus on the fundamentals, since you. You can't just Fourier analyze everything in sight.
Even if you don't want to think about black holes and the like, you should make an effort to understand the Unruh Effect: Observers in differentrelatively accelerated rest frames in flat Minkowski space will agree on the value of a quantum field at a point, but they will not agree on how many particles are in a system. The reality of particles depends on the representation of the algebra of observables.