A categorical characterization of ordinal numbers It's rather easy to notice that the operation of join of categories reproduces the ordinal sum once restricted to act on (iso classes of) well-ordered set; it's rather easy to see that $\alpha\star [0]$ (as a category) equals $\alpha+1$ (as number).
And in fact there is more ([Jec, p. 16]): the transfinite sequence $s\colon \mathbf{Ord}_{<\alpha}\to X\colon \langle a_\xi\mid \xi < \alpha\rangle$, extended with value $x_0$ on the successor $\alpha+1=\alpha^+$, corresponds exactly to the operation which, given a category $\cal C$, adds a (strained) terminal object $*$ defining $\mathcal C^\rhd = \mathcal C\star [0]$($=\mathcal C\coprod\{*\}$ with a unique arrow between any $C\in\cal C$ and $*$), and given a functor $F\colon \mathcal C\to \cal X$, it is extended with value $X_0\in\cal X$ on the unique object $*$ of $[0]$, coinciding with $F$ on $\cal C$.
Now: $\bf Cat$ endowed with the bifunctor $\star$ becomes monoidal [Joy, pp. 26-27] and the structure is closed on both sides (albeit not biclosed; the $\star$ is not symmetric)
Should this give a characterization of ordinals as some sort of cocompletion of the free $\star$-monoidal category on one generator? Can something non trivial be desumed by this fact?
Edit: in fact as it is stated the question is unclear. You must first complete, and then close under $\star[0]$ (I was confused, and maybe I still am, by the fact that $\star[0]$ commutes with colimits).
[Jec] Jech, Thomas. Set theory. Vol. 79. New York: Academic press, 1978.
[Joy] http://www.math.uchicago.edu/~may/IMA/Joyal.pdf
 A: You're not going to learn much about the conceptual or categorical structure of the ordinals from their classical presentation, not just because Excluded Middle is needed at every stage but because of normalformitis: the systematic elimination of structure.
That being said, there is a paper by Peter Johnstone, A topos-theorist looks at dilators (JPAA, 58, 1989) that applied Yves Diers' extensive work on categories with multiple-valued adjoints to Jean-Yves Girard's $\Pi^1_2$ Logic: Part I, Dilators  (Ann. Math. Logic 21, 1981). This studied ordinal-building constructions considered as functors that preserve pullbacks.
You cite Andre Joyal. He wrote a book with Ieke Moerdijk called Algebraic Set Theory (CUP 1995) and the subject was taken up by others and presented on Steve Awodey's website.
They characterised the universes of sets ($\in$-strutures in this context) and ordinals as free algebras for arbitrary unions and a unary operation $s$ (singleton or successor) subject to various equations: 


*

*for sets, no condition,

*for thin ordinals, $x\leq s x$,

*for plump ordinals, $x\leq y\Longrightarrow s x\leq s y$,

*for directed ordinals, $s x\lor s y= s(x\lor y)$,


using my terminology for the ordinals.
Earlier in the history of elementary toposes, Gerhard Osius wrote about Categorical Set Theory: a characterisation of the Category of Sets (JPAA, 4, 1974) and I picked up his ideas in my Intuitionistic Sets and Ordinals (JSL, 61, 1996).  My work was later taken up by Varmo Vene and others with application to process algebra.
In this setting, the membership relation is written $\in:X\to P X$ and so considered as a coalgebra for the powerset or other functors.  The important properties are extensionality and well-foundedness. The former is interpreted categorically by saying that the map $\in$ is mono and the second by a "broken pullback" diagram that you can find in my paper or my book.
The subset relation and the notion of simulation in process algebra are coalgebra homomorphisms. Recursion over ordinals was expressed by various people, including Adam Eppendahl, as coalgebra-to-algebra homomorphisms.
By applying these ideas in different categories, such as posets and binary semilattices, and with different notions of monomorphism, we obtain the four kinds of sets and ordinals above and others. These applications are summarised in Sections 6.3 and 6.7 of my book Practical Foundations of Mathematics (CUP 1999).
