A well-known example is the equivalence between groups and and one-object categories with all morphisms invertible (i.e. all morphisms are isomorphisms). The group is finite iff the category is.
A more exotic example of a finite category with all morphisms invertible is Conway's $M_{13}$ groupoid (see also this blog entry by John BaezJohn Baez, who is fond of most things categorical). Conway calls it "$M_{13}$" because there are $13$ objects each of whose automorphism groups is isomorphic with the Mathieu group $M_{12}$ (and any two objects are connected by $|M_{12}|$ isomorphisms).