Since the homotopy category depends only on the 2-skeleton of the simplicial set, the easiest thing to do is to take the 2-skeleton of an ∞-groupoid. For example, let $E$ be the nerve of the contractible groupoid on two objects. This has as a set of $n$-simplices the set $\{a,b\}^{[n]}$ of functions (of sets) from $[n]=\{0<...<n\}$ to the set $\{a,b\}$ (therefore the sequences of $n+1$ letters $a$ or $b$).
Then the 2-skeleton $\operatorname{sk}_2E$ of this simplicial set is just the subset of those functions $[n]\to \{a,b\}$ that factor via a non-decreasing map $[n]\to [2]$. This is not an $\infty$-category, however its homotopy category is the same as the homotopy category of $E$, and so it is exactly a contractible groupoid.