The question is in the title, but here is some background:
I previously asked for a general criterion to decide which colimits commute with which limits in the category of sets and received encouraging answers: (1) the question is already answered in some form by a paper of Foltz, (2) MO user Marie Bjerrum will soon provide what promise to be simpler to check criteria than Foltz's.
Now I'm curious about a special case of that question: for which categories $J$ do limits of shape $J$ commute with all filtered colimits? Certainly it is well known that finite $J$ satisfy this and it is also true that only filtered colimits commute with all finite limits [1], so one might initially guess that only finite limits commute with all filtered colimits. But this is wrong! For example, if $M$ is a finitely generated monoid (for example $\mathbb{N}$), limits of shape $BM$ commute with all filtered colimits (by $BM$ I just mean $M$ regarded as a one object category) [2]. A very similar argument actually shows that limits of shape $J$ commute with all filtered colimits as long as $J$ is "finitely generated", i.e., there is a finite set $F$ of morphisms such that every morphism in $J$ is a composition of some sequence of morphisms in $F$ (equivalently, there is a finite digraph and a functor from the free category on the digraph to $J$ which is surjective on morphisms). Are those all the $J$? EDIT: Answered: no! Since these work, any category with a finitely generated initial subcategory also works, and as Marc Hoyois said in his answer below, Paré proved that is all.
That is the end of the question but for the curious here are proofs of [1] and [2]:
[1] Assume colimits of shape $I$ commute with all finite limits in the category of sets. Let $F : J \to I$ be a finite diagram in $I$, we shall show that there is a cocone over it. Consider the functor $G : J^{\mathrm{op}} \times I \to \mathrm{Set}$ given by $G(j,i) = \hom_I(F(j),i)$. For any $i \in I$, $\lim_{j \in J^{\mathrm{op}}} G(j,i)$ is the set of cocones over $F$ with vertex $i$. If there were no cocones at all over $F$, $\mathrm{colim}_{i \in I} \lim_{j\in J^{\mathrm{op}}} G(j,i)$ would be empty. On the other hand $\mathrm{colim}_{i \in I} G(j,i) = \{\ast\}$ for any $j$ (because every element $f$ of $\hom_I(f(j),i)$ gets glued onto $id_{f(j)}$ by the map $G(j,f)$), so $\lim_{j\in J^{\mathrm{op}}}\mathrm{colim}_{i \in I} G(j,i) = \{\ast\}$.
Bonus mini-question: Is there a reference for the above result? I think it might be well known unwritten folklore.
[2] Let $F : I \to M\text{-Set}$ be a filtered diagram of sets with an action of the finitely generated monoid $M$. We want to show that $\mathrm{colim} _{i\in I}\;\mathrm{Fix}(F(i)) = \mathrm{Fix}(\mathrm{colim} _{i\in I}\; F(i))$ where $\mathrm{Fix}(X) = \{x \in X : x=mx$ for all $m\in M\}$. The canonical map from the LHS to the RHS is clearly injective (the notion of equality for fixed points in the colimit on the left is just that they are equal in the colimit of the underlying sets of the monoids, $F(i)$). If $M$ is generated by the finite set $m_1, \ldots, m_r$ we can see the canonical map is also surjective: given an element of the RHS represented by $x \in F(i)$, we know that for each $k=1,\ldots, r$, it is the case that $F(\alpha_k)(x) = F(\alpha_k)(m_k x)$ for some $j_k$ and some morphism $\alpha_k :i\to j_k$. Since $I$ is filtered we can equalize all the $\alpha_k$, and so for a single $j$ and a single $\alpha : i \to j$ we have that $F(\alpha)(x) = F(\alpha)(m_k x) = m_k F(\alpha)(x)$, so that $x$ comes from $F(\alpha)(x)$ in the LHS.
