There is an interesting analogy between primes in number fields and knots in 3-manifolds. This is can be explained by the analogy between Artin-Verdier duality theorem for number rings and the Poincare duality theorem for 3-manifolds. For a detailed discussion of various aspects of the origins of this analogy see Mazur's unpublished notes: "Remarks on the Alexander polynomial. Unpublished notes, 64, 1963", or the more recent book of Morishita: "Knots and primes: an introduction to arithmetic topology". One may also look at the recent lecture videos of Venkatesh at: https://www.math.arizona.edu/~swc/ on the topic.
I'm interested in learning more about the possible analogy between modular forms on the number-theoretic side, and topological analogs. Perhaps I need to look into the theory of topological modular forms, or perhaps topological quantum field theories (as Venkatesh suggests). Modular forms give rise to the theory of Hecke operators, Galois representations, and Selmer groups. The Galois representations should perhaps be defined on certain fundamental groups associated with link complements. These Selmer groups give us important information about L-functions, and can be studied from various perspectives, for instance, Iwasawa theory. What would be analog of this from the topological perspective, and where can I read about this theory, if it exists?
Of course, there is a clear analog of the Iwasawa polynomial in knot theory- the Alexander polynomial. The natural generalization would be to look for an analog for Selmer groups.
P.S. Morishita's book does go into Galois representations in the last chapter and the analog of deformation theory, but these do not seem to be explicitly related to TQFTs or toplogical modular forms as far as I can tell.