The idea of $\mathbb{F}_1$, the field with one element, might fit the spirit of the question. (If you feel it does not, please, let me know and I will remove the contribution.)
The idea that there should be a mathematical object resembling a finite field with a single element was put forward by Tits in the 1950s. Having a satisfactory theory of the 'field with one element' might allow progress on major mathematical problems (in number theory in particular), as it could allow to adapt the proofs of results, such as the Riemann-Hypothesis, known in the 'geometric case' (e.g., curves over finite fields), to the 'arithmetic case' (e.g., integers).
Various investigations related to this were and are undertaken. One could thus say there is mathematics on a concept without it being defined. For a recent contribution see for example Fun with $\mathbb{F}_1$ by Alain Connes, Caterina Consani, Matilde Marcolli.
