Most people regard induction and recursion as finitistic. As Andrej Bauer commented:
Induction and recursion are rules which tell you how to do something. They do not presuppose any kind of infinity or anything like that.
As a consequence, the standard weak metatheory used in practice is Primitive Recursive Arithmetic (PRA).
Although very weak, PRA is strong enough to formalize Gödel coding. Using such coding, the axioms of ZFC (to take a concrete example) can be enumerated and recognized by primitive recursive functions. One can also recognize formal proofs from ZFC axioms using a primitive recursive function.
Thus, there is a primitive recursive function $B_{ZFC}(x,y)$ which takes value $0$ if and only if the numeral $x$ is a Gödel code for a formal proof of the Gödel coded sentence $y$ from the axioms of ZFC. Taking $y$ to be the Gödel code for your favorite contradiction, say $0 = 1$. The consistency of ZFC can be expressed as $\forall x (B_{ZFC}(x,y) \neq 0)$. As Carl Mummert pointed out, this is a simple $\Pi^0_1$ statement in PRA. While we don't know whether it is true or false, it is a perfectly meaningful statement from the finitistic point of view.
