I asked myself exactly the same question a few weeks ago when I read post by John Baez in the $n$-Category Café on the Matthieu Group $M_{12}$ and groupoid $M_{13}$.

I don't think this can be done without more information about the vertex-set $X$.

In a sense a categorist like me is unqualified to answer this question because, just as we only known ordinary objects up to isomorphism, so we only know categories up to equivalence: a groupoid with a given vertex-set $X$ could equally well have any other set $Y$ for its vertices.

Presumably (at least, without loss of generality) we are looking for a transitive action, whilst the group should consist of the endomorphisms of one vertex. Then (in the finite case), the number of elements of $X$ must divide the order of the group, but the set $X$ was arbitrary.

The post by John Baez in the $n$-Category Café gives a nice example of how the the Matthieu Group $M_{12}$ can be presented more simply using a groupoid called $M_{13}$.