This may be more in line with Vincent R.B. Blazy's answer, but one thing I would like to add is the principle of inexhaustibility. This principle appears in Maddy's "Believing the Axioms. I", where it is stated as "the universe of sets is too complex to be exhausted by any handful of operations." For example, Maddy mentions it as justification for the axiom "an inaccessible cardinal exists":
... the universe of sets is too complex to be exhausted by any handful of operations, in particular by power set and replacement, the two given by the axioms of Zermel of and Fraenkel. Thus there must be an ordinal number after all the ordinals generated by [second-order] replacement and power set. This is an inaccessible.
However, this is not the case for the story of $\mathbb N$, in which there is a single simple operation exhausting $\mathbb N$. As in this way $\mathbb N$ is much more concretely generated, this may be seen as evidence that $\mathbb N$ is a completed infinite object, while $V$ is not. (I am not sure how to reconcile this with the formalism in Timothy Chow's answer, letting $\mathsf{ZF-Inf}+\lnot\mathsf{Inf}$ correspond to the maxim "$\mathbb N$ is not a completed infinity", because even then $V_\omega$ is exhausted by a handful of operations.)
Edit: Today I heard a relevant point phrased in a much more succinct way: "the intended model of Peano arithmetic $(\mathbb N,<,0,1,+,×)$ is intended to be minimal, while the intended model of set theory $(V,\in)$ is intended to be maximal."