This isn't an answer, so much as a string of hopefully-relevant observations:

Your question feels related to the distinction between analytic and synthetic mathematics. The former defines all objects and properties in terms of an existing theory, while the latter's core objects and properties are introduced as axioms. For example, analytic geometry is associated with Descartes - describing points, lines etc. in terms of co-ordinates and equations - and synthetic geometry with Euclid and Hilbert.

Personally, I associate the analytic approach with exploration: we find some examples of the things we're interested in, work out how to specify them (definitions in terms of an existing theory), study them as the concrete objects they are, and "pseudo-empirically" discover results which we then establish more reliably with proofs. Meanwhile, the synthetic approach takes some things that we already have a good feel for - in Euclid's case from everyday life, but these days more often from playing around analytically - and tries to extract their "essence" in the form of axioms so that we can prove results about them without worrying about concrete details of "implementation".

These associations ¹ are conceptual only, and not hard-and-fast rules. Still, they suggest a way to think about your question:

In the "explorative" analytic approach, we might happen upon one specification first, discover that it implies another result, and then when investigating that result in its own right find that it leads back to the original specification. Meanwhile, in the synthetic approach, if two formalizations using the same language are truly equivalent then they will have exactly the same consequences, and in some sense it's the consequences that "matter": we could if we wished take both as primitive, along with the machinery needed to translate between them, and then observe that our choice of primitives contains redundancies (but the choice of which to discard, if any, is arbitrary). I think this is what Rota was getting at.

¹ There are other lurking associations too, e.g. between analytic mathematics and (material) set theory or Platonism, and between synthetic mathematics and category theory or formalism. But of course any philosophical stance or background theory is compatible with both approaches.