What notions are used but not clearly defined in modern mathematics? 
"Everyone knows what a curve is, until he has studied enough mathematics to become confused through the countless number of possible exceptions."
Felix Klein

What notions are used but not clearly defined in modern mathematics?

To clarify further what is the purpose of the question following is another quote by M. Emerton: 

"It is worth drawing out the idea that even in contemporary mathematics there are notions which (so far) escape rigorous definition, but which nevertheless have substantial mathematical content, and allow people to make computations and draw conclusions that are otherwise out of reach."

The question is about examples for such notions.
The question was asked by Kakaz
 A: One of the most important contemporary mathematical concepts without a rigorous definition is 
quantum field theory (and related concepts, such as Feynman path integrals).  
Note: As noted in the comments below, there is a branch of pure mathematics --- constructive field theory --- devoted to making rigorous sense of this problem via analytic methods.  I should add that there is also a lot of research devoted to understanding various aspects of field theory via (higher) categorical points of view.  But (as far as I understand), there remain important and interesting computations that physicists can make using quantum field theoretic methods which can't yet be put on a rigorous mathematical basis.
A: The field with one element, $F_1$. 
Georges Elencwajg in http://mathoverflow.tqft.net/discussion/968/notions-used-but-not-rigorously-defined/#Item_0
A: In proof theory, the notion of a "natural well-ordering" comes up, but isn't (perhaps can't be) defined formally.
In a similar vein, I'm told that inner model theorists were proving results about "the core model" for decades without having a precise definition of what it was.
A: In response to Colin Tan's request (below), I have posted these remarks as the TCS StackExchange question "Do the undecidable attributes of P pose an obstruction to deciding P versus NP?" 

That a mathematical idea be "clearly defined" is itself an idea that perhaps could be more clearly defined ... one candidate for a more rigorous assertion is that a mathematical intuition be formally decidable.   Moreover, widespread intuitions that are eventually proved to be decidable versus undecidable have an illustrious history in mathematics.
These reflections lead to the suggestion this community wiki's question would be better-posed mathematically (and might perhaps be more useful too) if it were amended to read:"What intuitions are commonly embraced and/or have proved to be broadly useful, but nonetheless are formally undecidable, in modern mathematics?"One specific example that comes to mind is  Emanuele Viola's theorem, with its implication that the set of Turing machines {M} associated to P has no decidable runtime ordering.  Viola's proof of undecidability was eye-opening to me, and it has filled the valuable role of leading me to wonder "What else is out there?"

To show the utility of these reflections, Section 1.5.2 of Sanjeev Arora and Boaz Barak's well-respected textbook Computational Complexity: a Modern Approach is titled "Criticisms of P and some eﬀorts to address them".  I have often wished that Arora and Barak had written more on this theme.  With the help of Viola theorem, this wich becomes specific and rigorous: a section titled  "What properties of P are not decidable in modern mathematics?" 
No doubt many more examples of "undecidable intuitions of modern mathematics" could be posted, and it would be great fun to read other people's examples. However, it seems inappropriate to amend the topic of a community wiki in such a fundamental respect, and so I am posting this amended question as a suggested general "answer" instead.

Partially in response to Colin Tan's request (in the comments below), I have posted on TCS StackExchange the specific question "What is the proper role of verification in quantum sampling, simulation, and extended-Church-Turing (E-C-T) testing?".
More broadly, on Lance Fortnow's weblog, under the topic "75 Years of Computer Science", the question is raised 

"Do there exist languages $L$ that are recognized solely by those Turing machines in $P$ whose runtime exponents are undecidable? Can examples of these machines and languages be finitely constructed?" 

... but I am not (yet) prepared to post this as a MathOverflow and/or TCS StackExchange  question.  Thanks and appreciation are extended to Colin.
A: The notion of a solution concept in game theory. Although the most famous example of such---Nash equilibrium---is rigourously defined, as are several others (correlated equilibrium, rationalizability, sequential equilibrium, etc.), there is no satisfactory general definition of the type of object of which these are tokens. Indeed, the purported definition that appears in this Wikipedia article is, in a sense, as far from informative as it could be without incurring a type mismatch.
A: The notion of noncommutative set is used for the intuition as the noncommutative analog of a set, as the von Neumann algebras or the ${\rm C}^*$-algebras are for the measurable or topological spaces. But unlike these notions of noncommutative topological or measurable space which are well-defined in the operator algebras framework, the notion of noncommutative set is not (yet) (well-)defined. See the post: What's a noncommutative set?
A: The natural numbers!  We discuss them as if there were a "standard model" ${\bf N}=(\{0,1,2\ldots\},_,\cdot,<)$ that (by incompleteness) doesn't happen to have a recursively enumerable first-order theory, and then act like that's fine because we all know what ${\bf N}$ is.  Or do we?
Believing in a standard ${\bf N}$ doesn't seem much different than the belief that there's a standard universe of sets, where CH has a truth value that we don't happen to know.  These days there is at least partial (not universal) acceptance (e.g. multiverse theory) that there isn't a unique set universe.
So are natural numbers different?  Or is ${\bf N}$ itself not a well-defined concept?  Why does anyone think every sentence in the language of arithmetic has a truth value?  Even if one believes this for $\Pi_1^0$ sentences (every Turing machine halts or doesn't), why believe it at higher quantifier depth?  Why believe it when there are set quantifiers?
Anyway even if we could somehow get our hands on the complete first-order theory of ${\bf N}$ (aka true arithmetic), that theory still (by Löwenheim–Skolem) has infinitely many models in any given "true" universe of sets.
A: Infinitesimals are almost in this category.
Technically, calculus generally uses limits instead of infinitesimals. And there are logical systems (e.g. nonstandard analysis) in which genuine infinitesimals are rigorously defined. However, people find infinitesimals easier for intuition even in the context of the standard analysis. This type of infinitesimal reasoning generally then needs to be transformed into standard proofs.
A: I asked about Defining variable, symbol, indeterminate and parameter previously on MO, and did not get any satisfying answers for all these concepts.  The one exception is that of variable (and meta-variable) where Neel gave good pointers.
A: 'Applied Mathematics' is a much-used term in modern mathematics, but I've yet to find a universally-agreed upon definition. Given its use as a major category ('pure' vs 'applied') and repository of sundry generalizations ('non-rigorous','relevant', 'not deep', 'critical to science', etc.), surely a precise definition is in order.
In the MSC, there is only one MSC code with this phrase (00A69). Based on this, maybe 'Applied Mathematics' is a field of inquiry which is not important
A: To state that a mathematical assertion is morally correct or morally true seems to convey a significant amount of mathematical content.  This may indicate to the reader/audience that the assertion has every right to be true, even if it may not yet be proven.
See Eugenia Cheng's article discussing morality in mathematics.
A: I have three (somewhat related) examples:

*

*The notion of explicit construction. Seeking explicit constructions to replace non-constructive existence proofs is an old endeavor. Computational complexity offers, in some cases, formal definitions (constructions that can be dome in P or in polylog space.) But these definitions are slightly controversial. In any case people looked for explicit constructions before any explicit definition for the term explicit construction was known.


*The notion of effective bounds/proofs. There are many important problems about replacing a proof giving non effective bounds with a proof giving effective bounds. Usually I can understand a specific such problem but the general notion of effectiveness is not clear to me. (A famous example: effective proofs for Thue Siegel-Roth theorem.)


*Elementary proofs. I remember that finding elementary proofs for the prime number theorem was a major goal. I was told what this means many times and in a few of those I even understood. But the notion of "elementary" proof in analytic number theory remained quite vague for me.
A: I'm not sure how well this fits the bill, but in algebraic geometry and number theory, the notion of mixed motives is still undefined, although people have a fairly good idea of what properties they want the category of mixed motives to have.
A: Left/right derived functors. If $F$ is an additive functor from a category $A$ to another category $B$, then the left/right derived functors of $F$ go from $A$ to... where? Not to $B$ certainly, because this would require global choice on $A$ or break canonicity.
There seem to be solutions nowadays, with the notions of derived categories and anafunctors. Unfortunately, there seems to be no introductory text yet which would systematically develop homological algebra in a clean way, without cheating and speculating over one's head. I am more than glad to be proven wrong...
PS. This might be what Harry Gindi is referring to.
A: Not only is the notion of chaos not well-defined (cf. the answer of Gerry Myerson), but the same holds true for its opposite: there is no universally accepted definition of integrable system yet.
A: The notion of canonicity (with respect to maps and objects) has thusfar evaded attempts by mathematicians to formalize it.  If I remember correctly, Bourbaki tried to give it a definition based on some ideas of Chevalley, but, at least to my knowledge, it was deleted from later drafts of the Elements because it was not a particularly useful notion (or perhaps it just didn't work out.  There was a thread on MO asked by Kevin Buzzard about this particular section of Bourbaki, and maybe you could find more details there).  Jim Dolan more recently tried to give a definition of a canonical transformation between functors, but his notion is essentially that of a transformation that is natural when restricted to the core groupoid.  However, this doesn't really capture all of the cases that we want, and I don't know of any serious attempt to make use of the notion.  
A: The notion of a $q$-analogue in enumerative combinatorics.
A: There are several examples in set theory; the three I mention are related so I will include them in a single answer rather than three.

1) Large cardinal notion.

I have seen in print many times that there is no precise definition of what a large cardinal is, but I must disagree, since "weakly inaccessible cardinal" covers it. Of course, if you retreat to set theories without choice then there may be some room for discussion, but this is a technical point.
People seem to mean something different when they say that large cardinal is not defined. It looks to me like they mean that the word should be used in reference to significant sign posts within the large cardinal hierarchy (such as "weakly compact", "strong", but not "the third Mahlo above the second measurable") and, since "significant" is not well defined, then...
However, it seems clear that nowadays we are more interested in large cardinal notions rather than the large cardinals per se. To illustrate the difference, "$0^\sharp$ exists" is obviously a large cardinal notion, but I do not find it reasonable to call it (or $0^\sharp$) a large cardinal.
And large cardinal notion is not yet a precisely defined concept. A very interesting approximation to such a notion is based on the hierarchy of inner model operators studied by Steel and others. But their meaningful study requires somewhat strong background assumptions, and so many of the large cardinal notions at the level of $L$ or "just beyond" do not seem to be not properly covered under this umbrella.

2) The core model.

This was mentioned by Henry Towsner. I do not think it is accurate that we were proving results about it without a precise definition. What happens is that all the results about it have additional assumptions beyond ZFC, and we would like to be able to remove them. More precisely, we cannot show its existence without additional assumptions, and these additional assumptions are also needed to establish its basic properties.
The core model is intended to capture the "right analogue" of $L$ based on the background universe. If the universe does not have much large cardinal structure, this analogue is $L$ itself. If there are no measurable cardinals in inner models, the analogue is the Dodd-Jensen core model, and the name comes from their work. Etc. In each situation we know what broad features we expect the core model to have (this is the "not clearly defined part"). Once in each situation we formalize these broad features, we can proceed, and part of the problem is in showing its existence. 
Currently, we can only prove it under appropriate "anti-large cardinal assumptions", saying that the universe is not too large in some sense. One of the issues is that we want the core model to be a fine structural model, but we do not have a good inner model theory without anti-large cardinal assumptions. Another more serious issue is that as we climb through the large cardinal hierarchy, the properties we can expect of the core model become weaker. For example, if $0^\sharp$ does not exist, we have a full covering lemma. But this is not possible once we have measurables, due to Prikry forcing. We still have a version of it (weak covering), and this is one of the essential properties we expect.
(There are additional technical issues related to correctness.)
But it is fair to expect that as we continue developing inner model theory, we will find that our current notions are too restrictive. As a technical punchline, currently the most promising approach to a general notion seems to be in terms of Sargsyan's hod-models. But it looks to me this will only take us as far as determinacy or Universal Baireness can go. 

3) Definable sets of reals.

We tend to say that descriptive set theory studies definable sets of reals as opposed to arbitrary such sets. This is a useful but not precise heuristic. It can be formalized in wildly different ways, depending of context. A first approximation to what we mean is "Borel", but this is too restrictive. Sometimes we use definability in terms of the projective hierarchy. Other times we say that a definable set is one that belongs to a natural model of ${\sf AD}^{+}$. But it is fair to say that these are just approximations to what we would really like to say. 
A: So-called Stiff ODEs might qualify. In the literature one finds plenty of different attempts to define the notion of a stiff initial value problem for an ODE, some of them more, some less precise and they all try to capture the phenomenon of rapid step size decrease when numerically integrating some IVPs with explicit schemes whereas some implicit schemes do very well without slowing down significantly. In fact, some authors use this as the definition of a stiff IVP.
A: For a number of years, different authors were using different definitions of "chaos", but I think that has settled down now. 
"Quantum group" may be a good answer. If Wikipedia can be trusted on this issue, "In mathematics and theoretical physics, the term quantum group denotes various kinds of noncommutative algebra with additional structure. In general, a quantum group is some kind of Hopf algebra. There is no single, all-encompassing definition, but instead a family of broadly similar objects." 
A: In Leo Corry's book Modern Algebra and the Rise of Mathematical Structures
, he chronicles how mathematicians have tried to give a formal definition of structure via lattice theory, Bourbaki's set theoretic structures, and category theory. At least according to Corry, the concept is still elusive and not really captured by any of the attempts.
A: The set of equivalence classes of irreducible, smooth representations of a reductive $p$-adic group $G$ should be partitioned into finite subsets called $L$-packets.  Each $L$-packet should correspond to a Langlands parameter, but since this correspondence remains conjectural, $L$-packets are not defined in general.  In some important cases, one knows exactly what the $L$-packets are.  For example, if $G$ is a general linear group, then the $L$-packets are singletons.  For other groups, there are some properties that $L$-packets are believed to satisfy, but that's not a definition.
A: Surprised nobody mentioned fractal yet. (Chaos has been mentioned but the connection is tenuous.)
No satisfactory definition of fractal exists. Mandelbrot tentatively defined a fractal as a set whose Hausdorff dimension is strictly larger than its topological dimension. But this leaves out many sets that most people agree are fractals, and it's hard to extend to other objects (like measures)  that one also wants to consider as fractals.
Taylor defined a fractal as a set with coinciding Hausdorff and packing dimensions. His goal was to leave out too irregular objects (for which different concepts of fractal dimension may differ), but according to his definition any smooth object is a fractal, and clearly fractal sets such as Bedford-McMullen carpets are left out.
In applied fields, a fractal is often defined as a set having some kind of similarity: small parts are similar to the whole set, perhaps in a statistical or approximate sense. While many fractals arising in practice do enjoy this feature, this is still a very vague definition.
Some authors consider any set or measure in Euclidean space to be a fractal, when the goal is to study properties typically associated with fractal sets, such as Hausdorff dimension.
At the end of the day, there is agreement that giving a universal definition of fractal is impossible, yet it is a useful concept to have around, and people know a fractal when they see it.
A: Today, there are 101 papers on mathscinet using the notion of planar algebra, discovered by V. Jones in 1998.
The fundation of planar algebras theory is waiting for some detailed proved with all i's dotted and t's crossed.
See for example the post:  What's the detailed proof of "the composition of planar tangles is well-defined"?
A: The concept of turbulence is still vaguely or ill defined such as applied to too many phenomena. 
Examples from Is there a mathematically precise definition of turbulence for solutions of Navier-Stokes? and elsewhere


*

*In the Ptolemaic Landau–Hopf theory turbulence is understood as a cascade of bifurcations from unstable equilibriums via periodic solutions ([the Hopf bifurcation][2]) to quasiperiodic solutions with arbitrarily large frequency basis.

*According to [Arnold and Khesin][3], in the 1960's most specialists in PDEs regarded the lack of global existence and uniqueness theorems for solutions of the 3D Navier–Stokes equation as the explanation of turbulence.

*Kolmogorov suggested to study minimal attractors of the Navier-Stokes equations and formulated several conjectures as plausible explanations of turbulence. The weakest one says that the maximum of the dimensions of minimal attractors of the Navier–Stokes equations grows along with the Reynolds number Re.

*In 1970 Ruelle and Takens formulated the conjecture that turbulence is the appearance
of global attractors with sensitive dependence of motion on the initial conditions in the phase space of the Navier–Stokes equations ([link][4]). In spite of the vast popularity of their paper, even the existence of such attractors is still unknown. 

*Existence of energy cascades (eg. Big vortices feeding on smaller vertices). This reflects the physical notion that mechanical energy injected into a fluid is generally on fairly large length and time scales, but this energy undergoes a “cascade” whereby it is transferred to successively smaller scales until it is ﬁnally dissipated (converted to thermal energy) on molecular scales

*Von Karman: “Turbulence is an irregular motion which in general makes its appearance in ﬂuids, gaseous or liquid, when they ﬂow past solid surfaces or even when neighboring streams of the same ﬂuid ﬂow past or over one another.” 

*Hinze: “Turbulent fluid motion is an irregular condition of the ﬂow in which the various quantities show a random variation with time and space coordinates, so that statistically distinct average values can be discerned.”

*Chapman: “Turbulence is any chaotic solution to the 3-D Navier–Stokes equations that is sensitive to initial data and which occurs as a result of successive instabilities of laminar ﬂows as a bifurcation parameter is increased through a succession of values.”

*Criteria listed in McDonough's notes: 


*

*nonrepeatability (i.e., sensitivity to initial conditions);

*disorganized, chaotic, seemingly random behavior 

*extremely large range of length and time scales

*enhanced diﬀusion (mixing) and dissipation (both of which are mediated by viscosity at molecular scales)

*three dimensionality, time dependence and rotationality (hence, potential ﬂow cannot be turbulent because it is by deﬁnition irrotational);

*intermittency in both space and time.


A: Some people claim to have defined the concept of "closed form" fully and precisely.  Have they?
A: Maybe situation in Matroid theory where there is several strict axiomatization schemes but its equivalence is not easy to prove, is interesting here. Probably there should be some generalization which would tie this different approaches into one, more or less obvious notion. 
There is even terminology connected with that phenomenon by G.C.Rota Cryptomorphism http://en.wikipedia.org/wiki/Cryptomorphism 
A: Notion of calculability:

A function of positive integers is calculable only if recursive.

Calculable function ( in a objective meaning) as used in Church-Turing Thesis http://plato.stanford.edu/entries/church-turing/
A: I think that the widely-used concept of concepts being clearly defined is not clearly defined.
For example: How could one decide whether a single "concept" is "clearly defined" or not? If one could, I would argue that then all of modern mathematics would be  not clearly defined. Already the concept of a set seems to lack a precise definition, and even lack the possibility to be defined precisely. 
Being clearly defined therefore seems to me at best like a vague comparative notion. For example, we could say that we regard some concept as clearly defined if its definition is as clear as the definition of a set, whatever "as clear" means in this context... 
A: There is no definition of what a set is.
A: I could never find a both rigorous and universal definition of what exactly an L-function is, despite Selberg's introduction of the class bearing his name.
