One of the most important contemporary mathematical concepts without a rigorous definition is quantum field theory (and related concepts, such as Feynman path integrals).
Note: As noted in the comments below, there is a branch of pure mathematics --- constructive field theory --- devoted to making rigorous sense of this problem via analytic methods. I should add that there is also a lot of research devoted to understanding various aspects of field theory via (higher) categorical points of view. But (as far as I understand), there remain important and interesting computations that physicists can make using quantum field theoretic methods which can't yet be put on a rigorous mathematical basis.