The fact that you have to consider setiods shouldn't be too surprising, since if we're working ($n>0$)-categorically the distinction between sets and setoids isn't invariant under equivalence. If C and D are strict 0-categories (i.e., sets), then the equivalence relation is just the identity relation. If C and D are weak 0-categories (setoids), then $[C, D]$ is the weak 0-category (setoid) whose elements are the set functions $f : C \to D$, and where two functions $f, g: C \to D$ are equivalence if $f(c) \sim g(c)$ for all $c$ in $C$.
Sets equipped with equivalence relations have the simplest not-entirely-trivial homotopy theory I know. Larusson has a nice discussion of it.