I think there are many examples, spread out across a continuum of how "wrong" the definition really was. Of course, strictly speaking a definition cannot be "wrong", or can only be wrong in the logical sense of not umambiguously denoting a class of examples. E.g., Newton's and Leibniz's definition of the derivative was wrong -- or better, not well-defined! -- because it used infinitesimals in a way that was not formalized and could not be formalized in the context of known mathematics.

There are a lot of definitions that in retrospect look too limited or pedestrian: e.g., defining a manifold to be a certain kind of subset of Euclidean space. (Some people would say that the definition of a Riemann integrable function is "wrong" in this sense. I disagree -- the notion of Riemann integrability is a natural one that comes up e.g. in characterizations of uniform distribution of sequences.)

It seems like you are looking for examples of the following kind: the definition is given and then, in the same paper (or book, or whatever) a theorem is given using the definition. But contemporary mathematicians who look back at the theorem agree that the conclusion is not the desired one.

I can think of one instance of this, although it is of relatively minor importance. R.G. Bartle's 1955 paper *Nets and Filters in Topology* was one of the first to try to explicitly work out the folkloric "equivalence" between nets and filters when studying convergence on topological spaces. The way to do this is to, given a net on a topological space, associate a filter, and conversely, and then prove theorems about these associated nets and filters having the same convergence properties. But the definition Bartle gives of how to associate a net to a filter is "wrong", in the sense that certainly you want that when you in turn associate a filter to that net you get the filter that you started with, but his definition does not have this property (and the right definition does!). See for instance the last page of

http://alpha.math.uga.edu/~pete/convergence.pdf

for some more discussion of this.

In general, I would think that one has to be rather well-read in a subject area to come up with such examples, because -- thankfully! -- a truly "wrong" definition is usually swiftly drowned out by the correct definition.

21more comments