Let me set up a strawman:

One might entertain the following criticism of Godel's incompleteness theorem: why did we ever expect completeness for the theory of PA or ZF in the first place? Sure, one can devise complete theories semantically (taking all the statements that hold in some fixed model), but one has usually discovered something special (e.g. elimination of quantifiers) when a naturally framed theory just turns out complete.

Now perhaps one could defend Godel's theorem as follows:

By Godel, the theory of the standard natural numbers has no recursive axiomization, but equally remarkably PA has no recursive non-standard models (Tennenbaum's theorem). That means that the incompleteness of arithmetic has a deeper character than, say, the incompleteness of group theory -- there exhibiting groups with distinct first-order properties easily suffices.

My question:

Does there exist any sort of converse to Godel's incompleteness theorem. A converse might say that when one has incompleteness and also some reasonable side condition (I'm suggesting but not committed to "there exists only one recursive model"), then there must exist some self-reference mechanism causing the incompleteness? Or stronger perhaps, the theory must offer an interpretation of some sufficiently strong theory of arithmetic?