**Edit:** I just saw that you were willing to take "every epi split" as an assumption. In this case there is a considerably more classical approach:

**Lemma** A topos in which every epimorphism split is equivalent to the topos of sheaves on a complete boolean algebra.

This a relatively classical result. If I remember well it can be found in MacLane and Moerdijk "Sheaves in geometry and logic". SO if you simply add to this condition the usual "complete distributiviy" assumption on the lattice of sub-object of the terminal you can apply the result you quoted and you get a topos of sheaves over a complete atmoic boolean algebra, i.e. a slice of the category of sets.

I'm leaving the original answer below, which I think is also interesintg.

I'm proposing the following statement. I'm relatively sure it is true. But I have to admit that a full proof might require a bit of work to fill all the gaps. What I mean is that if you really need to use it somewhere, I wouldn't quote this post as a proof !

**Claim:** A completely distributive boolean topos is the topos of $G$-set for $G$ a groupoid.

So $G$-set for $G$ a groupoid is the next best thing to slice of the category of sets: it is a category that is locally a slice of the category of sets. Also the result mentioned in the question can essentially be recovered (without too much work) as a special case of this one.

I' don't think I need to explain what is a Boolean Topos, nor why this is a reasonable candidate to repalce boolean algebras. The tricky notion is "completely distributive".

For a regular cardinal $\kappa$, $\kappa$-distributive Grothendieck topos or $\kappa$-topos, is a $\kappa$-exact localization of a presheaf category. Usual formal arguement should implies that this is equivalent to the assumption that the "colim" functor from the category of small presheaves of the topos $Psh(\mathcal{T}) \rightarrow \mathcal{T}$ commutes to $\kappa$-small limits.
Note that commutation of this functor to finite limits is a known way to encode all of the usual Giraud's axioms of a topos except accessibility, see for example Lack-Garner

By totally distributive I mean $\kappa$-distributive for all $\kappa$, which seems to also be the definition of the papers mentioned by Mike Shulman in the comments.

$\kappa$-distributivity can also be rephrased in a way that is a little closer to distributivity of lattices: C.Espindola has some papers on infinitary categorical logic where he studied a bit these $\kappa$-toposes. He has observed that this condition of $\kappa$-distributivity on a topos can be rephrased as, informally:

"a $\kappa$-small transfinite composition of covering sieve is a covering sieve".

this has to be formalized using trees. i.e. given a $S$ a tree where each branch has height less than $\kappa$, and $D:S^{op} \rightarrow \mathcal{T}$ a functor such that for each node of the tree $s$, the childrens of $s$ form a covering of D(s) (a jointly epimorphic familly).
For each branch $b$ of the tree, you can form its limit $S_b$, then his condition is that for each such diagram, the $S_b \rightarrow S_0$ (with $S_0$ the root) form a covering of $S_0$.
For a topos this is equivalent to $\kappa$-distributivity.

As we are assuming this for all $\kappa$ there might be further simplification on this condition (and one can maybe get rid of these trees in this case) But I'm not sure about it at this point.

Now:

**Claim 2** A totally distributive Grothendieck topos is a presheaf topos.

This seems to follow from lemma 1 in the second paper linked by Mike, though it is late and I have some trouble following this paper that I discovered a few minutes ago only. The real reason I believe this claim is true is because of an unpublished result of C.Espindola in which I'm relatively confident and which implies this.

It is then a classical result (exercise ? ) that a presheaf topos $Prsh(I)$ is boolean if and only if $I$ is a groupoid. Hence the first claim follows from this second one.

Completely and totally distributive categories I, mta.ca/~rrosebru/articles/ctd.pdf, andAn adjoint characterization of the category of sets, mta.ca/~rrosebru/articles/accs.pdf $\endgroup$ – Mike Shulman Nov 16 '18 at 18:38Sketches of an Elephantsays that a localic topos is a slice category of Set if and only if it is atomic, and if and only if it is Boolean and has enough points. The localic condition can be removed for $\infty$-topoi (golem.ph.utexas.edu/category/2018/06/…) although then the notion of "slice category" is more general; also the notion of "atomic topos" is not very clearly related to complete distributivity. $\endgroup$ – Mike Shulman Nov 16 '18 at 18:41