Andreas Blass has already provided a good reference in the literature, but unfortunately I cannot read German, so I've had to make do with writing my own answer.
As you observed, you're clearly not going to get away from the abstract concept of 'collections of objects,' since it's pretty fundamental in mathematics, but I would argue that ordinals are not an intrinsically set-theoretic notion any more than, say, well-founded trees are. This isn't to say that these ideas aren't important in set theory, but I would say that if one were really committed to formalizing mathematics 'without sets,' eschewing ordinals or well-founded trees because of their applicability in set theory wouldn't really be a good idea.
It is entirely possible to give a relatively self-contained theory of ordinal-indexed lists of ordinals that is equiconsistent with $\mathsf{ZFC}$. I will sketch such a theory. Furthermore, I would argue that this theory is no more 'set-theoretic' than, say, second-order arithmetic formalized in terms of numbers and sequences of numbers.
The theory has two sorts $\def\Ord{\mathrm{Ord}}\Ord$ and $\def\Lst{\mathrm{Lst}}\Lst$ (for ordinals and lists of ordinals, respectively). I'll use the convention that lowercase Greek letters represent ordinals and lowercase Latin letters represent lists.
As primitive symbols, we have a constant $0$ in $\Ord$, a binary relation $\leq$ on $\Ord$, and two functions $(x,\alpha) \mapsto x_\alpha : \Lst \times \Ord \to \Ord$ and $\ell : \Lst \to \Ord$. $x_\alpha$ is meant to be the $\alpha$th element of the list $x$ and $\ell(x)$ is meant to be the length of the list $x$. We'll use the convention that if $\alpha \geq \ell(x)$, then $x_\alpha = 0$.
We have the following axioms and axiom schemas.
Linear order with least element: $\leq$ is a linear order on $\Ord$ with least element $0$.
List extensionality: $x = y$ if and only if $\ell(x) = \ell(y)$ and for all $\alpha < \ell(x)$, $x_\alpha = y_\alpha$.
List padding: For any $\alpha \geq \ell(x)$, $x_\alpha = 0$.
Successor: For every $\alpha$, there is a $\beta > \alpha$.
Supremum: For every $x$, there is an $\alpha$ such that $\alpha \geq x_\beta$ for all $\beta$.
Infinity (existence of a limit ordinal): There is an $\alpha > 0$ such that for all $\beta < \alpha$, there exists a $\gamma < \alpha$ with $\beta < \gamma$.
Well-ordering: For any formula $\varphi(\alpha,\bar{\beta},\bar{x})$ and parameters $\bar{\beta}$ and $\bar{x}$, if there is an $\alpha$ such that $\varphi(\alpha,\bar{\beta},\bar{x})$ holds, then there is a least such $\alpha$.
List comprehension: For any formula $\varphi(\alpha,\beta,\bar{\gamma},\bar{x})$, parameters $\bar{\gamma}$ and $\bar{x}$, and ordinal $\delta$, if $(\forall \alpha < \delta )\exists ! \beta \varphi(\alpha,\beta,\bar{\gamma},\bar{x})$, then there is a $y$ such that $\ell(y) = \delta$ and for each $\alpha < \delta$, $\varphi(\alpha,y_\alpha,\bar{\gamma},\bar{x})$ holds.
Given a list $x$ and an ordinal $\alpha$, write $x \ll \alpha$ to mean that $\ell(x) < \alpha$ and for all $\beta$, $x_\beta < \alpha$.
- List bounding: For any formula $\varphi(x,\alpha,\bar{\beta},\bar{y})$, any parameters $\bar{\beta}$ and $\bar{y}$, and any ordinal $\gamma$, there is an ordinal $\delta$ such that for all lists $x \ll \gamma$, if there is an $\alpha$ such that $\varphi(x,\alpha,\bar{\beta},\bar{y})$ holds, then there is an $\varepsilon \leq \delta$ such that $\varphi(x,\varepsilon,\bar{\beta},\bar{y})$ holds.
These axioms are far from optimal (for instance we could combine successor and supremum by requiring that $\alpha > x_\beta$ for all $\beta$), but I think they're a reasonably well-motivated set of principles for capturing the notion of collections of ordinal-indexed lists of ordinals that are 'complete' (in roughly the same way that models of $\mathsf{ZF}$ are 'complete').
It's fairly immediate that $\mathsf{ZF}$ (and therefore $\mathsf{ZFC}$) interprets this theory by looking at the class of ordinals and functions $x : \alpha \to \Ord$. Conversely, to show that this theory interprets $\mathsf{ZFC}$, we need to roughly do the following:
Show that well-ordering and list comprehension allow you to perform transfinite induction.
Use this to show that Gödel's pairing function on ordinals is definable. (Specifically, show that we can uniformly build lists giving arbitrarily long initial segments of the projections of the pairing function.)
Define the notion of ordinal computation (with, e.g., ordinal Turing machines). Use some standard work in that area to show that we can interpret $\mathsf{KP} + {V=L}$ with some defined predicate membership predicate on $\Ord$.
Show that the interpreted copy of $L$ is closed under full replacement (by list comprehension and supremum), and show that it is closed under (internal) power set (by list bounding). Conclude that $L \models \mathsf{ZFC}$.
One thing to note is that this procedure will not produce a bi-interpretation with $\mathsf{ZFC}$. In order to get a bi-interpretation, we would need some axiom asserting that we can 'list' all of the lists $x\ll\alpha$ for any $\alpha$, but this axiom actually seems a lot less natural to me in this context than the others I've listed. (In particular, it has a far less canonical flavor, since it's literally a form of the axiom of choice, and it requires a moderately complicated defined notion like the pairing function to formalize.) The sketchiest one is of course list bounding, but in some sense that corresponds nicely to the semi-frequent opinion that the sketchiest axiom of $\mathsf{ZF}$ is power set.
