What are "branes", and why do they form a category?

Question

I've been trying to read Kapustin–Witten - Electric–Magnetic Duality And The Geometric Langlands Program recently, as someone whose mathematical interests are in the Langlands program. I have some physics background, but not including string theory. I'm looking to understand "branes", which play a big part in the paper, and so far, in this context, here is my understanding (please feel free to correct any misconceptions):

Kapustin–Witten formulate a TQFT with gauge group $G$ on a spacetime of the form $\Sigma\times C$, where $C$ is going to play the role of the curve of the geometric Langlands correspondence. $\Sigma$ is a Riemann surface with boundary $\partial\Sigma$, and we are going to look at an effective field theory on $\Sigma$ after compactification on $C$. The resulting fields on $\Sigma$ are going to have to satisfy the Hitchin equations, whose solutions are parametrized by the Hitchin moduli space $\mathcal{M}_{H}(G,C)$. The assignment of the value of the field on $\Sigma$ is going to described by maps from $\Sigma$ into $\mathcal{M}_{H}(G,C)$. I believe this is what physicists refer to as a "sigma model".

As a side remark, $\mathcal{M}_{H}(G,C)$ has many fascinating properties, for example it is hyperkahler, and related to both $\operatorname{Bun}_{G}(C)$ and $\operatorname{Loc}_{G}(C)$, and its mirror pair concerns the Langlands dual ${}^{L}G$ in place of $G$. This makes it of interest in the Langlands program.

As far as I understand, a brane is a condition on where $\partial \Sigma$ gets sent to in $\mathcal{M}_{H}(G,C)$, i.e. a submanifold of $\mathcal{M}_{H}(G,C)$. There are actually two TQFT's in consideration here, an "A-model" and a "B-model", and for the A-model the submanifold $\partial \Sigma$ can get sent to is a Lagrangian (or more generally coisotropic) submanifold, while for the B-model it is a complex submanifold.

But for some reason it is more than just what I described — in fact, a brane in the B-model (or a B-brane) really consists of a coherent sheaf. Hence the appearance of the derived category of coherent sheaves in the formulation of homological mirror symmetry.

My first question is: Why are B-branes coherent sheaves? Apparently this is related to some sort of "boundary modification" (see 5.2 of Frenkel's survey Gauge Theory and Langlands Duality), but I don't know what these are, and I would be happy to see an explanation or elaboration.

Furthermore, branes (both A-branes and B-branes) form a category. There is supposed to be some sort of physical intuition for the morphisms between branes, but it is not clear to me what that is. For A-branes, the corresponding category is supposed to be some enlargement of the Fukaya category, whose morphisms are Floer chain groups.

This is my second question: What is the "physical intuition" behind morphisms of branes, why are they the Floer chain groups for A-branes, and why do branes form a category?

Morphisms of branes are actually pretty important in another way, as this is how Kapustin–Witten get from A-branes to D-modules on $\operatorname{Bun}_{G}(C)$ (in terms of which the geometric Langlands correspondence is stated). This is constructed by means of a "canonical coisotropic A-brane" (which is $\mathcal{M}_{H}(G,C)$ itself considered as an A-brane) whose endomorphisms are supposed to give a sheaf differential operators after some sort of localization or sheafification process. Kapustin's notes Lectures on Electric–Magnetic Duality and the Geometric Langlands Program say this is related to "insertion of vertex operators" (I don't really understand what this means). This also somehow looks reminiscent of Beilinson–Drinfeld's work producing D-modules (see 9.2 of Frenkel - Lectures on the Langlands Program and Conformal Field Theory) although as far as I know that makes no reference to branes.

Finally, my third question: Why do endomorphisms of the canonical coisotropic A-brane give us differential operators?

Answer 1

Let me start by putting your questions into a bit more context. Kapustin and Witten's story occurs within string theory, a theory of 1-dimensional extended objects. Strings may be "closed," forming loops, or "open," with two boundary points. As a string evolves in time, it traces out a 2-dimensional "worldsheet," which is your $\Sigma$. Moreover, string theory is a quantum theory, and so the state of a string is given by a vector in a vector space, and the dynamics of the string are determined by summing over all possible trajectories (worldsheets) the string can take.

Now, what are (Dirichlet) branes in string theory? For closed strings, all you need to specify their dynamics is a target space for them to move around in, which for you is $\mathcal{M}_H(G, C)$. However, for open strings with two endpoints, one must also specify boundary conditions. These boundary conditions are given by branes, and constrain the dynamics so that the string endpoints may only move about inside the brane.

One is free to choose different boundary conditions (branes) for the left and right ends of the string! Thus, given two branes $X$ and $Y$, we may consider the dynamics of strings with one endpoint on $X$ and the other on $Y$. The states of such a string form a vector space. Moreover, given three branes $X, Y$, and $Z$, we can consider the physical process in which strings stretching from $X$ to $Y$ and from $Y$ to $Z$ join along $Y$ to produce a string stretching from $X$ to $Z$. Thus, we have a composition map, and we may define a linear category whose objects are branes, and whose morphism spaces are vector spaces of open string states.

Now, there are a few caveats needed to make this quite correct. First of all, string states are subject to gauge redundancies, which by a standard procedure (BV-BRST) means that that "vector spaces of string states" are really chain complexes, whose cohomology encode the "physically meaningful" states. Moreover, composition is only associative up to the gauge redundancy, and rather than a category, we obtain an $A_\infty$ category, where the $n$-ary composition is defined by the physical process in which $n$ strings join into one.

All of this makes sense in an arbitrary string theory, but your questions are more specifically in the context of topological string theory. Topological string theory is a topological shadow of more physical string theories, defined by severly restricting the dynamics of the strings, and focusing only on the space of physical ground states (i.e., states of zero energy).

In the A-model, strings are restricted to move along holomorphic curves, with their boundaries on Lagrangian branes. Thus, the objects are Lagrangian branes, and the morphisms are the space of string ground states: there is a potential ground state at each point of intersection (since the distance between the branes is proportional to the energy of a string stretched between), but these ground states may quantum tunnel into each other via dynamical processes (holomorphic curves). To answer your second question: The quantum-corrected space of ground states of A-model strings stretched between Lagrangian branes is the Floer homology.

The B-model is more restrictive than the A-model: the only allowed dynamics of the string are trivial (i.e., the only allowed worldsheets are constant maps). However, the boundary conditions in the B-model are more interesting: they are given by complex subvarieties of any dimension. There is actually a bit more data: the endpoint of the string is a charged particle, and couples to a dynamical gauge field living on the brane. This gauge field is given by a holomorphic vector bundle over the complex subvariety, whose dimension is the "number" of branes sitting along that subvariety. The space of ground states stretching between the branes is the space of sheaf morphisms; for example, the morphisms for a stack of $N$ branes intersecting a stack of $M$ branes transversely at a point is given by $Hom(\mathbb{C}^N, \mathbb{C}^M) \cong \mathbb{C}^{NM}$, since one must choose which of the $N$ or $M$ branes the string starts or ends on.

A general coherent sheaf is basically just a complex vector bundle over a subvariety, but if you want a more precise picture, recall that a coherent sheaf is locally given by the cokernel of a morphism $E \to F$ of complex vector bundles. The physical picture is that $F$ is a stack of branes, while $E$ is a stack of anti-branes, and the morphism denotes a possible value for the "tachyon field" valued in the vector bundle $Hom(E, F)$. The tachyon denotes a physical process by which the branes and anti-branes can partially annihilate, leaving behind the cokernel: a coherent sheaf. Thus, A coherent sheaf is the most general result of brane/anti-brane annihilations in the B-model, taking into account their gauge bundles. For the record, this process is also why D-brane charges are given by K-theory classes: virtual differences between vector bundles.

For your third question, I don't think I can give a complete answer, but one thing to say is this. While before I said that A-branes were given by Lagrangian submanifolds, this is actually too restrictive. In fact, the real condition is that the submanifold be coisotropic. Moreover, just as with B-branes, A-branes also should have gauge fields (bundles) living on them. For Lagrangian branes, these bundles must be flat, and in particular we may turn them off (take them to be trivial). However, for coisotropic branes of higher dimension, the A-model topological constraint actually prevents us from fully turning off the gauge field.