There are several examples in set theory; the three I mention are related so I will include them in a single answer rather than three.
- Large cardinal notion.
I have seen in print many times that there is no precise definition of what a large cardinal is, but I must disagree, since "weakly inaccessible cardinal" covers it. Of course, if you retreat to set theories without choice then there may be some room for discussion, but this is a technical point.
People seem to mean something different when they say that large cardinal is not defined. It looks to me like they mean that the word should be used in reference to significant sign posts within the large cardinal hierarchy (such as "weakly compact", "strong", but not "the third Mahlo above the second measurable") and, since "significant" is not well defined, then...
However, it seems clear that nowadays we are more interested in large cardinal notions rather than the large cardinals per se. To illustrate the difference, "$0^\sharp$ exists" is obviously a large cardinal notion, but I do not find it reasonable to call it (or $0^\sharp$) a large cardinal.
And large cardinal notion is not yet a precisely defined concept. A very interesting approximation to such a notion is based on the hierarchy of inner model operators studied by Steel and others. But their meaningful study requires somewhat strong background assumptions, and so many of the large cardinal notions at the level of $L$ or "just beyond" do not seem to be not properly covered under this umbrella.
- The core model.
This was mentioned by Henry Towsner. I do not think it is accurate that we were proving results about it without a precise definition. What happens is that all the results about it have additional assumptions beyond ZFC, and we would like to be able to remove them. More precisely, we cannot show its existence without additional assumptions, and these additional assumptions are also needed to establish its basic properties.
The core model is intended to capture the "right analogue" of $L$ based on the background universe. If the universe does not have much large cardinal structure, this analogue is $L$ itself. If there are no measurable cardinals in inner models, the analogue is the Dodd-Jensen core model, and the name comes from their work. Etc. In each situation we know what broad features we expect the core model to have (this is the "not clearly defined part"). Once in each situation we formalize these broad features, we can proceed, and part of the problem is in showing its existence.
Currently, we can only prove it under appropriate "anti-large cardinal assumptions", saying that the universe is not too large in some sense. One of the issues is that we want the core model to be a fine structural model, but we do not have a good inner model theory without anti-large cardinal assumptions. Another more serious issue is that as we climb through the large cardinal hierarchy, the properties we can expect of the core model become weaker. For example, if $0^\sharp$ does not exist, we have a full covering lemma. But this is not possible once we have measurables, due to Prikry forcing. We still have a version of it (weak covering), and this is one of the essential properties we expect.
(There are additional technical issues related to correctness.)
But it is fair to expect that as we continue developing inner model theory, we will find that our current notions are too restrictive. As a technical punchline, currently the most promising approach to a general notion seems to be in terms of Sargsyan's hod-models. But it looks to me this will only take us as far as determinacy or Universal Baireness can go.
- Definable sets of reals.
We tend to say that descriptive set theory studies definable sets of reals as opposed to arbitrary such sets. This is a useful but not precise heuristic. It can be formalized in wildly different ways, depending of context. A first approximation to what we mean is "Borel", but this is too restrictive. Sometimes we use definability in terms of the projective hierarchy. Other times we say that a definable set is one that belongs to a natural model of ${\sf AD}^{+}$. But it is fair to say that these are just approximations to what we would really like to say.