Abstract treatment of multivariate calculus relevant for optimization

Question

After studying the basics of (convex) optimization, I've become convinced there's sometimes a conceptual benefit in thinking of quantities like gradients etc. in a coordinate-free way, and keeping track of the spaces they naturally come from (e.g., while oftentimes we pick a basis $\mathbb{R}^n$ and think of the gradient as a vector also in $\mathbb{R}^n$, one can define gradients abstractly and see they naturally come from the dual $V^*$; another example would be optimizational duality).

I'm looking for a text that emphasizes such abstract points of view on the aspects of multivariate analysis relevant to (convex) optimization (no upper limit to how abstract (e.g. categories are fine)... I guess). Suggestions that are not obviously related to (convex) optimization are also welcome.

Answer 1

Optimization is also done in Banach spaces - don't know if this is abstract enough, but, see, e.g.

Barbu, Viorel, and Teodor Precupanu. Convexity and optimization in Banach spaces. Springer Science & Business Media, 2012.

There is also the classical text

Luenberger, David G. Optimization by vector space methods. John Wiley & Sons, 1969.

It seems like to following quote from the introduction is in line with your needs:

Some readers may look with great expectation toward functional analysis, hoping to discover new powerful techniques that will enable them to solve important problems beyond the reach of simpler mathematical analysis. Such hopes are rarely realized in practice. The primary utility of functional analysis for the purposes of this book is its role as a unifying discipline, gathering a number of apparently diverse, specialized mathematical tricks into one or a few general geometric principles.

You may also like the following intriguing and embarrassingly simple illustration from Luenberger's book of the idea of duality in optimization:

In words: The minimal distance of a point to a convex sets is equal to the maximum distance of said point to any hyperplane separating the point and the set. So minimizing distances to a set of points is equal to maximizing distances to a set of hyperplanes (in certain situations, of course…).

Answer 2

The following may be too basic for you, but here is some relevant material that I found useful as a physicist.

You may be interested in Div, grad, and curl are dead, by Burke. He was killed in a car accident before the book was formally published, but you can find PDFs by googling. The basic motivation is to present sophomore vector calculus using notation that doesn't obscure the underlying symmetries (e.g., the way the right-handedness of the definition of the curl obscures the parity-invariance of what we use it to describe). Another book by Burke is Spacetime, geometry, cosmology. This has a lot of material that relates to gradients, covectors, constraints, duality, and the kind of applications referred to in Dirk's answer and the Amari paper linked to in a comment. For example, someone who had read this book would be immunized against doing some of the ugly/wrong things Amari does, like equating a vector to a covector.

Burke used and further developed an older system of geometric representations originally due to Schouten. There is a 1954 book, Ricci-Calculus: An Introduction to Tensor Analysis and its Geometrical Applications, by Schouten that presents the geometrical stuff in detail, including a mystic mandala diagramming the relations between the different types of tensors (Hodge duals, ...). It also formalizes the physicist's way of describing types of tensors by their transformation properties. Schouten uses coordinate-based index notation, but the geometrical portion is coordinate-independent.

You might also want to learn about Penrose's abstract index notation, which is a coordinate-free notation that looks superficially like coordinate-based concrete index notation. It has the advantage that (1) it's more practical than "mathematician" notation for many complicated calculations of the type that come up in general relativity, and (2) when you want to take a result and apply it in a basis, the translation to concrete index notation is trivial. The clearest exposition I know of is in Penrose's pseudo-popular book The road to reality.

Abstract index notation also has a graphical version that naturally leads to an interpretation in terms of braids. This sort of thing has been developed by Cvitanovic and others, and a relevant keyword is "birdtracks." Cvitanovic has an online book on birdtracks.

Answer 3

You might appreciate: Multivariate Statistics: A Vector Space Approach, Morris Eaton, described here:

My interest in the coordinate-free approach to linear statistical problems was motivated by at least two things: first, a predilection for elegant mathematics applied to statistics and second, the hint that such an approach could beneficially be brought to bear on multivariate analysis. My first foray into coordinate-free multivariate analysis consisted of the approach to variance (MANOVA) models. Emboldened by this initial success, I set out to develop and extend multivariate analysis via a coupling of coordinate-free methods and invariance (group theoretic) arguments.

Answer 4

I can even top Dirk's reference:

Constantin Zălinescu, MR1921556, Convex analysis in general vector spaces, World Scientific Publishing Co., Inc., River Edge, NJ, 2002.

(Although I draw the line at metric spaces...)