I asked myself exactly the same question a few weeks ago when I read
<a href="http://golem.ph.utexas.edu/category/2013/02/m13.html">post by John Baez
in the <i>$n$-Category Caf&eacute;</i></a>
on the Matthieu Group $M_{12}$ and groupoid $M_{13}$.

I don't think this can be done without more information about the vertex-set $X$.

In a sense a categorist like me is unqualified to answer this question because,
just as we only known ordinary objects up to isomorphism,
so we only know categories up to equivalence:
a groupoid with a given vertex-set $X$ could equally well have any other set $Y$
for its vertices.

Presumably (at least, without loss of generality) we are looking for a transitive action,
whilst the group should consist of the endomorphisms of one vertex.
Then (in the finite case), the number of elements of $X$ must divide the order of the group,
but the set $X$ was arbitrary.