The question is in the title, but here is some background:

I previously asked for a general criterion to decide which colimits commute with which limits in the category of sets and received encouraging answers: (1) the question is already answered in some form by a paper of Foltz, (2) MO user Marie Bjerrum will soon provide what promise to be simpler to check criteria than Foltz's.

Now I'm curious about a special case of that question: for which categories $J$ do limits of shape $J$ commute with all filtered colimits? Certainly it is well known that finite $J$ satisfy this and it is also true that only filtered colimits commute with all finite limits [1], so one might initially guess that only finite limits commute with all filtered colimits. But this is wrong! If $M$ is a finitely generated monoid (for example $\mathbb{N}$), limits of shape $BM$ commute with all filtered colimits (by $BM$ I just mean $M$ regarded as a one object category) [2]. I think the same argument can actually show that limits of shape $J$ commute with all filtered colimits as long as $J$ is "finitely generated", i.e., there is a finite set $F$ of morphisms such that every morphism in $J$ is a composition of some sequence of morphisms in $F$ (equivalently, there is a finite digraph and a functor from the free category on the digraph to $J$ which is surjective on morphisms). Are those all the $J$?

That is the end of the question but for the curious here are proofs of [1] and [2]:

[1] Assume colimits of shape $I$ commute with all finite limits in the category of sets. Let $F : J \to I$ be a finite diagram in $I$, we shall show that there is a cocone over it. Consider the functor $G : J^{\mathrm{op}} \times I \to \mathrm{Set}$ given by $G(j,i) = \hom_I(F(j),i)$. For any $i \in I$, $\lim_{j \in J^{\mathrm{op}}} G(j,i)$ is the set of cocones over $F$. If there were no cocones at all over $F$, $\mathrm{colim}_{i \in I} \lim_{j\in J^{\mathrm{op}}} G(j,i)$ would be empty. On the other hand $\mathrm{colim}_{i \in I} G(j,i) = \{\ast\}$ for any $j$ (because every element $f$ of $\hom_I(f(j),i)$ gets glued onto $id_{f(j)}$ by the map $G(j,f)$), so $\lim_{j\in J^{\mathrm{op}}}\mathrm{colim}_{i \in I} G(j,i) = \{\ast\}$.

Bonus mini-question: Is there a reference for the above result? I think it might be well known unwritten folklore.

[2] Let $F : I \to M\text{-Set}$ be a filtered diagram of sets with an action of the finitely generated monoid $M$. We want to show that $\mathrm{colim} _{i\in I}\;\mathrm{Fix}(F(i)) = \mathrm{Fix}(\mathrm{colim} _{i\in I}\; F(i))$ where $\mathrm{Fix}(X) = \{x \in X : x=mx$ for all $m\in M\}$. The canonical map from the LHS to the RHS is clearly injective (the notion of equality for fixed points in the colimit on the left is just that they are equal in the colimit of the underlying sets of the monoids, $F(i)$). If $M$ is generated by the finite set $m_1, \ldots, m_r$ we can see the canonical map is also surjective: given an element of the RHS represented by $x \in F(i)$, we know that for each $k=1,\ldots, r$, it is the case that $F(\alpha_k)(x) = F(\alpha_k)(m_k x)$ for some $j_k$ and some morphism $\alpha_k :i\to j_k$. Since $I$ is filtered we can equalize all the $\alpha_k$, and so for a single $j$ and a single $\alpha : i \to j$ we have that $F(\alpha)(x) = F(\alpha)(m_k x) = m_k F(\alpha)(x)$, so that $x$ comes from $F(\alpha)(x)$ in the LHS.