The following may be too basic for you, but here is some relevant material that I found useful as a physicist.
You may be interested in Div, grad, and curl are dead, by Burke. He was killed in a car accident before the book was formally published, but you can find PDFs by googling. The basic motivation is to present sophomore vector calculus using notation that doesn't obscure the underlying symmetries (e.g., the way the right-handedness of the definition of the curl obscures the parity-invariance of what we use it to describe). Another book by Burke is Spacetime, geometry, cosmology. This has a lot of material that relates to gradients, covectors, constraints, duality, and the kind of applications referred to in Dirk's answer and the Amari paper linked to in a comment. For example, someone who had read this book would be immunized against doing some of the ugly/wrong things Amari does, like equating a vector to a covector.
Burke used and further developed an older system of geometric representations originally due to Schouten. There is a 1954 book, Ricci-Calculus: An Introduction to Tensor Analysis and its Geometrical Applications, by Schouten that presents the geometrical stuff in detail, including a mystic mandala diagramming the relations between the different types of tensors (Hodge duals, ...). It also formalizes the physicist's way of describing types of tensors by their transformation properties. Schouten uses coordinate-based index notation, but the geometrical portion is coordinate-independent.
You might also want to learn about Penrose's abstract index notation, which is a coordinate-free notation that looks superficially like coordinate-based concrete index notation. It has the advantage that (1) it's more practical than "mathematician" notation for many complicated calculations of the type that come up in general relativity, and (2) when you want to take a result and apply it in a basis, the translation to concrete index notation is trivial. The clearest exposition I know of is in Penrose's pseudo-popular book The road to reality.
Abstract index notation also has a graphical version that naturally leads to an interpretation in terms of braids. This sort of thing has been developed by Cvitanovic and others, and a relevant keyword is "birdtracks." Cvitanovic has an online book on birdtracks.