I asked myself exactly the same question a few weeks ago when I read post by John Baez in the $n$-Category Café on the Matthieu Group $M_{12}$ and groupoid $M_{13}$.
I don't think this can be done without more information about the vertex-set $X$.
In a sense a categorist like me is unqualified to answer this question because, just as we only known ordinary objects up to isomorphism, so we only know categories up to equivalence: a groupoid with a given vertex-set $X$ could equally well have any other set $Y$ for its vertices.
Presumably (at least, without loss of generality) we are looking for a transitive action, whilst the group should consist of the endomorphisms of one vertex. Then (in the finite case), the number of elements of $X$ must divide the order of the group, but the set $X$ was arbitrary.