Most harmful heuristic? What's the most harmful heuristic (towards proper mathematics education), you've seen taught/accidentally taught/were taught? When did handwaving inhibit proper learning?
 A: "Generalization for the sake of generalization is a waste of time"
I think that generalization for the sake of generalization can be rather fruitful.
A: "Differentiation and integration are inverse operations."
To many calculus students, this is their conception of the fundamental theorem. There's truth to this heuristic, of course, but one needs to be constantly informed by a much deeper understanding of integration (and differentiation) in order to properly wield this correspondence in most situations beyond those encountered in a first course in calculus.
A: "you'll need a computer for that".
A: Two bad principles that taste worse together: Decimals are the true numbers.  Rounding makes no difference.
Since students learn about decimals after they've learned about whole numbers and fractions, they might assume that decimals are always the preferred way to represent real numbers, and so everything should be converted to decimals.  Meanwhile, since in generally one cannot be expected to write out an infinite decimal expansion, they might assume that stopping after two decimal places makes no difference.
I'm not saying that approximations are bad.  But it's bad to approximate if you have no sense of your error tolerance, or even of the fact that you're introducing an error at all.
Here are two perverse outcomes.


*

*Imagine a problem whose answer is, say, $\pi/4$, and a solution that ends like this: $$\text{blah blah blah} = \pi/4 = 3.14/4 = .785.$$  I'm sure that there are some situations where it's important to know that your answer is between $.78$ and $.79$.  But much of the time, conversion to decimals obscures what's going on.

*(Small sample size alert!)  About half of my calculus students will, on the first day of class, mark the equation $\frac{1}{3} = 0.33$ as ``true''.

A: Linear algebra purely as row manipulations. I've written about this here:

Students stuck in a rut of thinking of
  matrices as a clever way to arrange
  numbers will get lost and confused; I
  know this because I was one of those
  students. I had to “de-program” what I
  was taught in high school before I
  could grasp what was going on.

A: "Truth is binary. If a theorem has been proven once, there is no need in a second proof."
A: From Keith Devlin's article
http://www.maa.org/devlin/devlin_06_08.html
"Multiplication is repeated addition."
This is true when multiplying natural numbers, but is a special case of a scaling operation in the reals. We know it is also a rotation in the complexes, but that should probably be left out at the beginning, although it might interesting to think about how one would include them at the beginning.
Devlin also mentions "exponentiation is repeated multiplication."
A: Similar to Tom's answer, 

a vector is a mathematical quantity with both a magnitude and a direction.

Useful for distinguishing between speed and velocity but little else. The above is a typical definition from a physics textbook I had on the shelf; here in British Columbia, vectors are introduced in high school physics but not high school math.  By the time students get to linear algebra in first- or second-year university, it can be hard to convince them that a real number (much less a polynomial) can be a vector. Usually, you have to resort to "a real number does too have a direction: positive or negative" and even then they don't believe you because 

a scalar is a mathematical quantity with a magnitude and no direction

and so if real numbers are vectors, how can they be scalars? 
Don't even ask about function spaces.
A: One extremely harmful heuristic I held until fairly recently: identifying math with algebraic manipulation.  When asked to prove an identity or an inequality I would often dive straight into algebraic manipulation of the relations that I knew, wasting many many hours of my time.  I have found that it is much more useful to try and test statements against examples I already know, and to try and rephrase identities and inequalities in terms of a statement in natural language that I have some intuition for.
A: 
The "size" of a finite-dimensional vector space is proportional to its dimension.

In fact, the "size" of a finite-dimensional vector space is almost always better thought of as being exponential in its dimension. This is easiest to see for (finite-dimensional) vector spaces over finite fields, which have finite cardinality. But it's a better heuristic even for vector spaces over infinite fields.
Internalizing the correct intuition makes it clear why forming the (algebraic) direct product of two vector spaces causes their dimensions to add, and not to multiply as you might naively expect based on the fact that taking the direct product of groups multiplies their orders.
Another confusing point to which this misconception leads regards the advantage that quantum computers give over classical ones. The difference is sometimes stated as "quantum computers have a state space that's exponentially large in the number of qubits," but this is highly misleading, because classical computers also have a state space that's exponentially large in the number of bits. The better intuition is: since quantum computers have a state space whose dimension is exponentially large in the number of qubits, the state space itself is actually doubly exponential in the number of qubits, while the state space of a classical computer is only singly exponential in the number of bits.
The reason why this misconception is so widespread is that early courses in linear algebra almost always begin with vectors spaces over infinite fields (usually $\mathbb{R}$ or $\mathbb{C}$), which have infinite cardinality, so the dimension is the only finite number available. This practice leads to misleading intuition for general vector spaces.
A: "Categories can be specified by objects alone."  It's easy to get this impression, because people who are familiar with the categories in question already know the morphism structure, and don't bother to specify it.  There is a related heuristic concerning the composition law, but it doesn't seem to burn people as often.
A: "Mathematical knowledge is contained and communicated primarily by documents."
I'm not sure if this is a heuristic, but in terms of beliefs that inhibit learning, this is definitely the one that hurt my mathematical development the most. 
I would say the correct statement is "Mathematical knowledge is contained primarily in the minds of mathematicians and communicated primarily by informal oral communication."
This problematic belief grew out of the way that I (and pretty much everyone else) was taught mathematics at the undergraduate and beginning graduate level. In this setting texts are a central authority and a complete, well-written resource for the knowledge needed to solve any mathematical problem encountered.
In the world of mathematical research, this is no longer the case. I finally figured this out by reading Thurston's essay "On proof and progress in mathematics", which I would strongly recommend for any beginning mathematician. 
Maybe it is possible to do research mathematics using papers as a primary resource, but I believe this is highly inefficient. I spent several years trying to learn the noncommutative standard model by reading the available papers on the subject and made no real progress. Looking back, I don't think I ever had a chance of succeeding with this approach.
I would guess that to be successful in mathematics, it is absolutely vital to become regularly involved in conversations with working mathematicians, as awkward and intimidating as that might be.
A: Talking about "functions" when we are actually talking about equivalent classes of functions almost everywhere equal
An element of a $L^p(X)$ space is usually called a "function", and is usually denoted by letters that are used typically for functions ($f$, $g$, $h$, etc.). 
It seems to be a harmful heuristic to act "as if" $L^p(X)$ is made of functions, as   a function is really something that should give you a value for each point $x$ in $X$. I am aware that it is now common practice, but I am sure it would help to actually introduce an actual name besides "function" to call "equivalent classes of functions modulo equality almost-everywhere". This concept is fundamental, and should be given a proper name. I don't have a proposition for such name, but I kind of wish someone in the past did.  
A: The following refers to the school system in Germany, it may be different in other countries:
In my opinion, one really bad heuristic happens in elementary school, when children learn arithmetics with natural numbers. They learn that addition and subtraction are two entirely different things, because they are taught $a+b=b+a$ but $a-b\neq b-a$. Thus addition is commutative, and subtraction is not. At that level, numbers are solely understood as enumerations of objects.
Then they learn about numbers with units, such as lengths or prices or weights. Also they learn that numbers might have geometric meaning, e.g. as lengths of line segments. But still no concept of negative numbers.
Years later, when they finally get to know negative numbers as well, they have so much incorporated that subtraction is something different from addition that they have difficulties to grasp that $a-b=a+(-b)=(-b)+a$, i.e. that subtraction is nothing else than addition of a negative number.
I think that postponing negative numbers so long is a mistake, and that children in elementary school would very well be capable to understand them.
A: "A continuous function is one you can draw without raising the pencil"
This has terrible disadvantages when generalizing functions defined on a real interval to non connected sets, non compact sets and in general topological spaces.
A: That there is something weird and unsavory about field extensions that are not separable and that serious contemplation of such things should be put off to the indefinite future.
(In fact, much of the richness and "pathology" of geometry in characteristic p is easily understood once one has a firm grasp of how field extensions behave.)  
A: In elementary school, there are false principles which take a lot of effort to overcome:


*

*Math problems have one answer.

*There is one right method.


These may be ok (though the second is debatable) when you are working on $1+2$, but not when you are supposed to isolate a variable, to graph a function, to recognize how you can apply the chain rule, to solve a complicated word problem, or to prove something. Many students don't think math is a place to experiment or to apply creativity. They are afraid to take incorrect steps even when it is no longer convenient or possible to say what the right first step is.
There is an interesting app called Dragonbox. It is very popular in Norway. When children think of algebra as a puzzle or game, they feel free to experiment, and they quickly learn to do things like isolate variables which usually give algebra students trouble. See also Terry Tao's blog posts on gamifying algebra. Students can learn to solve the problems, but have difficulty because these incorrect principles get in the way.
A: "Basic (and useful) mathematics is about calculations and higher (pure) mathematics is about proof."
One reason I think this is harmful is that there is no sharp line between calculations and proofs. Very often a certain calculation is essentially the proof except for a few logical connectives. Conversely, in formal logic, one can create a "calculus" that makes proofs appear to be calculations.
Another reason is that it leads students (and more importantly teachers!) to think that a drastic change of mindset is required to learn higher mathematics.
It is indeed true that analysis is quite different from calculus even though there is strong linkage. However, the former also leads to better techniques to calculate things. Putting too much emphasis on the "proof" aspect of analysis tends to put off a lot students who enjoyed playing with polynomials, trigonometry and calculus. Conversely, many students who like to work with proofs are encouraged to believe that what they are doing is somehow "superior" (higher) to "mere" calculus; they then do not do enough computational drills which ill-serves them if they actually take up mathematics!
A: The opposite of Qiaochu's dictum is just as misleading - "formulas are functions".  There are a lot of non-denoting expressions!  It's just that mathematicians don't tend to write non-denoting terms very often.  Of course, there's a good reason for that - you can't prove anything interesting about non-denoting terms (or rather, way too much).  But then students never get the intuition that there are expressions which are 'junk', nor tools to prove that something is 'junk'.
My favourite 'junk' expression is
$$1/\frac{1}{\left( x - x \right) } $$
Lest you think this is not very important, try to "teach" first-year calculus to a computer, and you'll see how these non-denoting terms are most troublesome.
A: "Vectors are directed line segments." When worded this way, this utterance is only acceptable if the student is satisfied with getting on his or her bicycle at the end of class and never returning to mathematics again. 
A: Not sure if this qualifies exactly, but I can never remember which theorems of group theory apply to finite groups, and which ones apply to groups in general. Anytime I remember a result, I have this sinking feeling that it appears in a textbook preceded by "for the remainder of this section, let G be a finite group." I'm not sure how well-founded this fear is (other than the theorems that obviously don't make sense for infinite groups, like the Sylow theorems).
A: Not the most harmful, but a fun example (credit due to Tony Varilly): 

"You can't add apples and oranges."

False. You can in the free abelian group generated by an apple and an orange. As Patrick Barrow says, "A failure of imagination is not an insight into necessity."
A: 
A natural (iso)morphism is one that is "canonical", or defined without making "choices", or that is defined "in the same way" for all objects.

This is a heuristic I found in every introductory text on category theory I can remember reading (and usually followed with the single/double dual of a vector space as an example) and it took me quite a while to realize that this is not only inaccurate, but just plainly wrong. 
Explanation of "wrongness": A natural morphism is a morphism between two functors. That is, a morphism in the category of functors between two categories. And as such, should be thought as usual as mapping the "data" in a way that preserves the "structure" and choices have really nothing to do with it. 
For example, thinking of a group $G$ as a one object category, functors from it to the category of sets form the category of $G$-sets. A morphism of $G$-sets is a map of sets preserving the action of $G$ and not a map of sets that "does not involve choices". Same goes for other familiar categories of functors (representations, sheaves etc.)
Another example is the category of functors from the one object category $G$ again to itself. To give a natural map (isomorphism) from the identity functor of $G$ to itself is just to pick an element of the center of $G$. I don't imagine anyone describing it as doing something that "doesn't involve choices".
Moreover, every category $C$ is the category of functors from the terminal one-object-one-morphism category to $C$. Hence, every morphism in any category is a "natural morphism between functors" so there is really no point in specifying a heuristic for when a morphism is "natural". This is utterly meaningless.
In the other direction, it is easy to write down "canonical" object-wise maps between two functors that fail to be natural in the technical sense. Conisder the category of infinite well ordered sets with weakly monotone functions. The "successor function" is definitely defined "in the same way" for all objects, but is not a natural endomorphism of the identity functor in the technical sense.
Explanation of harmfulness": Well I guess it is clear that a completely wrong heuristic is a bad one, but I'll just point out one specific example that is perhaps not so important, but shows clearly the problem. When showing that every category is equivalent to a skeletal category there is a very "non-canonical" construction of the natural isomorphisms. I saw several people get seriously confused about this.
Some thought:  One might argue that this heuristic was advanced by the very people who invented category theory (like Maclane) and thus, it is perhaps a bit presumptuous to declare it as "plainly wrong". My guess is that at the time people where considering mainly large categories (like all sets, all spaces, all groups etc.) as both domain and codomain of functors and were focusing on natural isomorphisms. In such situations it is unlikely that the functor will have non trivial automorphisms (or have very few and "uninteresting" ones) and therefore a natural isomorphism will be in fact unique so maybe this is the origin of the heuristic (It is just a guess, I am not an expert on the history of category theory). 
This relates to the point that by definition, if specifying an object does not involve choices, then it is unique (this is a tautology). So when we say that an isomorphism is "canonical" we usually mean that given enough restrictions, it is unique (and not just natural in the technical sense). For example, the reason we identify the set $A\times (B \times C)$ with the set $(A\times B)\times C$ is not because there is a natural isomorphism between them, but because if we consider the product sets with the projections to $A,B$ and $C$, then there is a unique isomorphism between them. And this is in line with the general philosophy of identifying objects when (and only when) they are isomorphic in a unique way. In contrast, we don't identify two elements of a group $G$, just because they are conjugate (This is "naturally isomorphic" viewed as functors of one object categories $\mathbb{Z}\to G$) precisely because this natural isomorphism is not unique.
Well, I did not intend this to get so lengthy... I was just anticipating some "hostile" responses defending this heuristic, so I tried to be as convincing as possible!
A: Almost any heruistic can be "most harmful" if used by a teacher in a situation when
the audience does not know why it makes sense, and without an explanation. This is especially dangerous in the 
frequent case that the heruistic  does not actually seem reasonable to a person seeing it for the first time, since it 
makes sense only in some ways but not others. It might require months of experience 
for an uninitiated person to understand how and why it applies.
For example, the heuristic of schemes as manifolds is such -- every algebraic geometer
understands it, but it actually is harmful to a person who is seeing schemes for a first
time (such a person would vary likely interpret this heruistic as saying that affine 
schemes are trivial to understand). Same applies to "integration is the inverse of
 differentiation", and some of the other answers to this question.
Of course, these heuristics are also the most useful ones, once you (and any audience you might have) actually understand them.
The whole point of learning math is to gain more such heuristics, and to makes the ones you have
more precise. For this reason, it seems to me that the use of such heruistics on an
unprepared audience is the most common problem in the lectures by the very best mathematicians.
A related problem is the an abundance of statements that are not strictly true, but
"correct in spirit". Again, this may be very useful in research or when talking to a person of appropriate sophistication, but it is very bad for students if such statements are used carelessly and without explanation.
P.S. This whole answer is generalization for the sake of generalization. Was it a waste of time, I wonder?
A: Division by Zero is Infinity. This was taught in my seventh grade while the teacher was explaining the concept of infinity and it's definition(s).
False: division by zero is Undefined.
A: This isn't really a heuristic, but I hate "functions are formulas". For most students it takes a really long time to think of a function as anything other than an algebraic expression, even though natural algorithmic examples are everywhere. For example, some students won't think of
\begin{gather}
f(n) = \{\text{1 if $n \bmod 2 = 0$ $\lor$ $-1$ otherwise}\}
\end{gather}
as a function until you write it as $f(n) = (-1)^n$
A: Also not really a heuristic, but "differentiation is easy," as encoded in the following two sub-heuristics:


*

*Differentiation is just repeated application of the product and chain rules, and

*Most functions are differentiable most of the time.


Edit:  Someone doesn't seem to like this answer, so I'll expand.  Students who leave calculus with this impression enter analysis with a disadvantage: differentiation is not a property that "most" functions have in any reasonable sense, not even continuous ones, and to compute the derivative of a function that isn't given as a sum of compositions of "elementary" functions requires an entirely different mindset than the one that values the product and chain rule.
A: I wish to point the attention on Pete Clark's very relevant initial comment. The term heuristic is often taken as synonymous to non-rigorous method, only based on intuition or experience. I personally dislike this acceptance of the word in mathematics, and I suspect it is not even historically correct (now I'm curious to check the use of it in the classic authors). The etymology of the adjective, from the verb εὑρίσκω (to find, discover) means "aimed to find". As I see it, it is exactly the method we follow when looking for a solution of a problem: using all implications of being a solution in order to identify a candidate solution. Of course, the heuristic is only half the job, and it is only rigorous if followed by part 2: checking the solution. But there's a very smart idea in it. For instance: solving an equation, transform it, but do not check the equivalence of each single step, just follow a chain of implications. So, what is harmful is not the heurstic method, but leaving out the (often less creative) part 2. That said, here's my example: let F be a smooth function bounded below (or a functional) with only one critical point. Then one would argue:

Any minimum point of F(x)=0 satisfies
  F'(x)=0, whose only solution is
  x0. Hence, x0 is
  the minimizer.

False!, if one does not check that F(x0)≤F(x) for all x ("direct method in Calculus of Variations") or if one has not proved the existence of a minimizer (indirect method). Many students make this mistake... but not only them!
A: Any attempt to draw a fat Cantor set is a bad heuristic in my opinion.  I saw such a diagram as an undergrad and believed for a while that there were intervals contained in the fat Cantor set.  I don't think it's possible to express in a picture that a fat Cantor has positive Lebesgue measure and has empty interior.
A: 
A tensor is a multidimensional array of numbers that transforms in the following way under a change of coordinates...

I saw that for years, and I never understood it until I saw the real definition of a tensor.

[Clarification]
Sorry, I did leave that very vague.  A tensor is a multilinear function mapping some product of vector spaces $V_1\times \cdots \times V_n$ to another vector space.  In the context of differential geometry, we're really talking about a tensor field, which assigns a tensor to every point that acts on the tangent and/or cotangent spaces at the point.

A more abstract definition is possible by considering tensor products of vector spaces, but the definition using multilinear functions is (to me) extremely intuitive and general enough for a first encounter.  It also leads naturally enough to the abstract concepts anyway, as soon as you start thinking about the set of all tensors of a particular rank and its structure.
The "multidimensional array" definition suffers from conflating object and representation.  The array is an encoding of the underlying multilinear function, and it's perfectly reasonable if understood in that way (to partially reply to Scott Aaronson's comment).  Unfortunately, the encoding depends on an arbitrary choice (coordinate system), while the underlying function obviously doesn't, so it gets very confusing if you try to use it as the definition.
Regarding accessibility (also referring to Scott Aaronson's comment): I don't really agree: I think multilinear functions are pretty accessible.  Assuming a familiarity with vector spaces and linear transformations, multilinear functions are a natural and very tangible extension of those ideas.  And since multilinearity is the key concept underlying tensors, if you're going to deal with tensors, you should really just bite the bullet and deal with the concept.
A: "Teach the subject before its applications."
Some important constructions seem quite pointless until you understand the rationale for them. For example, I recall finding the lectures in freshman linear algebra on constructing Jordan Normal Form extremely boring and pointless until JNF came up in the context of solving linear ODEs a year later. "That's what Jordan Normal Form is for!" - I thought - "I wish I knew that a year ago!"
A: Along the same lines as Qiaochu's and Zach's responses, the commonly taught heuristics pertaining to functions, differentiability and integration are a pet hate of mine.
I certainly left school thinking of functions as formulas involving combinations of elementary functions and having a very poor understanding of the relevance and correct relationship between integration and differentiation, the worst manifestation of which, now that I'm a bit older, seems to have been that 
Differentiation is a nice, computable operation and tells you about functions; integration is hard and tells you about areas under curves.
Areas under curves never seemed interesting. As an analyst, my personal feelings towards them are now almost entirely reversed and I think of integration as my friend and differentiation as the enemy. 
Differentiation uses up regularity; integration smooths. 
A: The "FOIL" (first+outside+inside+last) mnemonic for multiplying two binomials is terrible.  It suppresses what is really going on (three applications of the distributive property) in favor of an algorithm.  In other words, it is teaching a human being to behave like a computer.
The legacy of FOIL is clear when you ask your students to multiply three binomials, or two trinomials.  Students usually either have no idea what to do, attempt it but get lost in the algebra, or succeed but complain about the arduousness of the task.
A: Writing a proof as a chain of expressions connected by equals signs whether they are appropriate or not.  
A: "Stacks are schemes with groups attached to points."
I don't know how much damage this has caused, but I never understood how it was actually helpful to anybody. Not only is it hand-wavy (which is okay for a heuristic), but it's hand-wavy in a way that can't really be corrected (because it's false). My feeling is that people who adopt this heuristic are trapped. If they use the heuristic to come up with a result, it's very hard to sharpen the reasoning to turn it into a proof. You have to just start from scratch and not use the heuristic.
A: Two-column proofs
Usually the only proofs that students see upon graduating from high-school are the geometry "two-column" proofs, and trying to convince them that the essence of mathematical proof lies not in the form but in the logical deductive argument takes a lot of convincing. 
A: 
In linear algebra, things that are specified by a single number are scalars and things that are specified by a collection of multiple numbers are vectors (or higher-rank tensors).

This is wrong for at least two reasons. First, it blurs the distinction between a one-dimensional vector space over a field and field itself. Second, and perhaps more problematically, it gives the incorrect impression that (e.g.) if $\vec{V}(\vec{r}) = (V_x, V_y, V_z)$ is a vector field, then the individual component $V_x(\vec{r})$ is a scalar field and transforms accordingly under coordinate rotations.
A: The excluded middle ( A Law or an Heuristic) .
On a more general level given any closed question: Is it A or B ? , the heuristic says it is one or the other disregarding the option : the question is wrong or stupid or irrelevant or incomplete.
The principle of excluded middle disregards intuitionist logic. And has been harmful in not providing direct (constructive) proofs which are often more clear - yet can be harder to find.
Intuitionism is is also rather natural : being against anti-communists does not means you are a communist.
A: Perhaps one of the worst heuristics is Cramer's rule as a method of computing determinants (a hideous sum over $n!$ signed permutations ...) in linear algebra classes. (I don't know why it is even mentionned.) So often have I seen students (correctly) compute the determinants of $3 \times 3$ matrices and then take only 6 permutations to try to calculate the determinants of $4 \times 4$ matrices.
