# Coend of $\mathscr{D}(F(\bullet), G(\bullet))$

(Crosspost from stack)

Given categories $\mathscr{C}$ and $\mathscr{D}$ and functors $F,G: \mathscr{C} \to \mathscr{D}$, we can form a bifunctor $$\mathscr{D}(F(\bullet), G(\bullet)): \mathscr{C}^\text{op} \times \mathscr{C} \to \mathsf{Set}$$ and the end of this functor is the set of natural transformations from $F \Rightarrow G$. (I guess we need $\mathscr{C}$ to be small in order to guarantee there is a set of such natural transformations, in general.)

Can I say anything about the coend? Is it some familiar thing?

• The question is already interesting when we take $F, G$ to be identity functors. See for example ncatlab.org/nlab/show/trace+of+a+category – Todd Trimble Sep 25 '16 at 6:30
• I think the comment of @მამუკა ჯიბლაძე in the answer below is what I was referring you to in your SE post. – Daniel Robert-Nicoud Sep 25 '16 at 10:45
• Todd's comment would qualify as an answer, I guess. What is the trace of the category of finite-dimensional vector spaces? finitely generated $R$-modules? – HeinrichD Sep 25 '16 at 11:48
• If you take $F=G=1$ and $\mathcal C=\mathbf{Vect}_K$ then $\int^V \hom(V,V)\cong K$, right? – Fosco Sep 25 '16 at 13:28
• @HeinrichD, I think this qualifies as a proof that $\int^V \hom(V,V)\cong K$. Indeed, $\hom(V.V)\cong V^*\otimes V$, which in turn is isomorphic to $\hom(V,K)\otimes V$. But now this is only a fancy way to write $\text{Lan}_11(K)\cong K$. – Fosco Sep 25 '16 at 13:29

I'm not sure I have much to add beyond my comment, but I might add a point of view (which could in turn provide some search terms).

I'd situate this construction within the bicategory of (small) categories, profunctors/bimodules, and transformations between them. Recall that a profunctor $R: C \nrightarrow D$ is a functor of the form $C^{op} \times D \to Set$ (conventions may differ), composed much in the way relations are, via the formula

$$(C \stackrel{R}{\nrightarrow} D \stackrel{S}{\nrightarrow} E)(c, e) = \int^{d: D} R(c, d) \times S(d, e).$$

(If you like, you can consider the bicategory of profunctors as biequivalent to a strict 2-category whose objects are small categories and whose morphisms $C \to D$ are given by cocontinuous functors $Set^C \to Set^D$.)

This bicategory is compact closed in an evident bicategorical sense: we have a symmetric monoidal bicategory whose tensor at the object level is given by cartesian product of small categories, and each object $C$ has a monoidal dual given by the opposite category $C^{op}$. For each $C$, the unit $\eta_C: 1 \nrightarrow C^{op} \times C$ is given by $\hom_C: C^{op} \times C \to Set$. (In the cocontinuous functor picture, it's the unique (up to isomorphism) cocontinuous functor $Set \to Set^{C^{op} \times C}$ that takes the terminal object $1$ to $\hom_C$.) The counit $\epsilon_C: C \times C^{op} \nrightarrow 1$ may also be described by a hom-functor, but it is probably more illuminating to think of it in terms of the cocontinuous functor picture, given by taking the coend $\int^C: Set^{C^{op} \times C} \to Set$.

Since we are working in a compact closed (bi)category, we can expect certain resonances with constructions in other compact closed categories, such as the category of finite-dimensional vector spaces. The construction in question is a profunctor composite

$$1 \stackrel{\eta_C}{\nrightarrow} C^{op} \times C \stackrel{F^{op} \times G}{\nrightarrow} D^{op} \times D \stackrel{\epsilon_{C^{op}}}{\nrightarrow} 1$$

which is certainly akin to trace operations in linear algebra. Thus, in linear algebra over a field $k$, we have the notion of trace of an endomorphism $f: V \to V$, which we can form categorically as the composite:

$$\text{Tr}(f) = \left(k \stackrel{\eta_V}{\to} V^\ast \otimes V \stackrel{1 \otimes f}{\to} V^\ast \otimes V \stackrel{eval_V}{\to} k \right)$$

where the first map $\eta_V$ takes $1 \in k$ to $\sum_{i = 1}^n f^i \otimes e_i$ (here $\{e_1, \ldots, e_n\}$ is a basis of $V$ and $f^i$ is the dual basis; the expression $\sum_{i=1}^n f^i \otimes e_i$ is independent of basis). Similarly, we speak of the trace of an endoprofunctor $B: C \nrightarrow C$; after a brief Yoneda-lemma type calculation, one finds that the composite

$$1 \stackrel{\eta_C}{\nrightarrow} C^{op} \times C \stackrel{1 \otimes B}{\nrightarrow} C^{op} \times C \stackrel{\epsilon_{C^{op}}}{\nrightarrow} 1$$

is the profunctor $1 = 1^{op} \times 1 \to Set$ taking the unique object of $1$ to $\text{Tr}(B) = \int^{c: C} B(c, c)$.

Thus we could also describe your construction as the trace of the endoprofunctor or endobimodule $B: C \nrightarrow C$ defined by $B(c, d) = \hom_D(Fc, Gd)$. Possibly this gives a useful search term.

Usually such traces are challenging to calculate explicitly (for example, determining the trace of an identity functor can be nontrivial). Among the properties of trace formally deducible from compact closed structure is $\text{Tr}(B \circ B') \cong \text{Tr}(B' \circ B)$.

• I didn't know this nice description of profunctors as cocontinuous functors. When we replace $Set$ by $\{0,1\}$, we should get a similar description of relations $C \looparrowright D$: These are maps of sup-lattices $P(C) \to P(D)$, right? – HeinrichD Sep 25 '16 at 20:02
• Yep! This also works generally where $C, D$ are posets and $P(C) = [C^{op}, 2]$ is the sup-lattice of down-sets, provided we use "relation" to mean one where $R(a, b) = \text{true}$ and $a' \leq a$ implies $R(a', b) = \text{true}$ and $b \leq b'$ implies $R(a, b') = \text{true}$. (Usually I call such relations "bimodules".) – Todd Trimble Sep 25 '16 at 20:19

Basic rule in general category theory: Restrict to categories with one object, aka monoids, and see what happens. (Alternatively, restrict to thin categories, aka preorders.) Gain some intuition and generalize this to arbitrary categories.

So here, if $M$ is a monoid, we have two objects $X$ and $Y$ of $\mathscr{D}$ on which $M$ acts from the left via morphisms of monoids $F:M \to \mathrm{End}(X)$ and $G:M \to \mathrm{End}(Y)$. Then, the set $\mathrm{Hom}(X,Y)$ carries a left $M$- and a right $M$-action. The coend identifies these two actions in a universal way: It is given by $\mathrm{Hom}(X,Y) / (G(m) \circ f \sim f \circ F(m))_{f \in \mathrm{Hom}(X,Y),\, m \in M}$. If the action $F$ is trivial, this is a generalization of the set of coinvariants. Namely, this is the special case $\mathscr{D}=\mathsf{Set}$ and $X=1$. Dually, notice that the end of $\mathrm{Hom}(X,Y)$ is the set of morphisms $f \in \mathrm{Hom}(X,Y)$ which "believe" that the two actions agree, i.e. $G(m) \circ f = f \circ F(m)$ for all $m \in M$, and that this generalizes the set of invariants when the action $F$ is trivial.

Now, if $\mathscr{C}$ is any small category and $F,G : \mathscr{C} \to \mathscr{D}$ are functors, you may view these as "multi-object" actions of $\mathscr{C}$ on $\mathscr{D}$. The coend of $\mathrm{Hom}(F(-),G(-))$ is the quotient set of the coproduct $\coprod_{x \in \mathscr{C}} \mathrm{Hom}(F(x),G(x))$ which identifies the two actions of $\mathcal{C}$. This means that we identify, for every morphism $m : x \to y$ in $\mathscr{C}$ and every morphism $f : F(y) \to G(x)$ in $\mathscr{D}$ the two morphisms $G(m) \circ f : F(y) \to G(y)$ and $f \circ F(m) : F(x) \to G(y)$ in the coproduct.

• Probably relevant: coend of $F(\_)\times G(\_)$ (for contravariant $F$ and covariant $G$) is $F\otimes_{\mathscr C}G$ – მამუკა ჯიბლაძე Sep 25 '16 at 6:52