There are books or articles that deal with generalizations of functional analysis in the sense that the inner product need not be positive-definite or that works with functions over manifolds?

I've never seen any literature on indefinite Hilbert spaces. But there is a huge amount of work done on spaces of functions and distributions on manifolds - just about any reasonable textbook on operator theory should discuss this.
–
Paul SiegelMar 21 '13 at 12:36

1

I was under the impression that indefinite Hilbert spaces show up in some formulations of quantum field theory, in connection with gauge invariance. Unfortunately, this impression in my mind doesn't seem to be attached to any references.
–
Andreas BlassMar 21 '13 at 12:42

4

I guess what you are looking for are Krein spaces and their (better behaved) special cases, Pontryagin spaces. Heinz Langer and his school have worked a lot on them.
–
Delio MugnoloMar 21 '13 at 13:46

3

Infinite dimesional spaces with indefinite metrics have been studied by the Ukrainian school of operator theory (Krein). See, for example, "Hilbert space with an indefinite metric" by Nikolskii.
–
jbcMar 21 '13 at 13:48

also useful might be to look at a book on "Indefinite linear algebera...." by I. Gohberg et al.; but yes, since spaces with indef. IPs are called Krein spaces, I guess jbc's and Delio's comments above are the ones to chase!
–
SuvritMar 21 '13 at 16:18