Perhaps this question overlaps with similar ones, ... but I want to focus on a particular possible cause of confusion. I notice that students are often confused by the concepts of "infinite" and "unbounded". So, when asked if the set of invertible matrices is compact, they reply "no, because there are an infinite number of matrices with non zero determinant, therefore the set is unbounded". Actually this happens in Italian, where the corresponding words ("infinito" and "illimitato") are almost synonyms in everyday language. Does this happen in English too, or other languages?. I wonder: what if we chose another name for the two concepts? Would they make this mistake anyway? One way to check this would be to compare with what happens in other languages, where perhaps the words chosen do not create the confusion. Do you have other examples of this situation? Can you suggest different math concepts which in one language are named with synonyms, but not in another? Do you know if this problem has been studied anywhere?

"Open" and "closed". Every reasonable human being on the planet, who has not studied topology, will assume that something can either be open or closed, but not both. This often causes students to make statements like "Set A is open, therefore it is not closed, thus ..." 


My favourite is this: some books use "completely reducible" for semisimple and "irreducible" for simple. As a result, every irreducible module is completely reducible. 


If you count "or" as a mathematical concept, the the fact that it is fundamentally inclusive in mathematics but often exclusive in most other uses of English can lead students to mistakes. 


I'm surprised nobody has mentioned "onetoone function" for injection and "onetoone correspondence" for bijection. 


My favourite example is "complex analysis" (as well as "complex" and "imaginary" numbers). Students, mostly in advance, feel it too complex. There should be probably a better name but it's too late to change... 


The article "Surprises from mathematics education research: student (mis)use of mathematical definitions" by Edwards and Ward addresses some of your concerns in the context of U.S. undergraduates. From the introduction:



It's not just students who get confused by terminology. I was recently puzzled for quite a while until I realised that finite von Neumann algebras can be infinitedimensional. 


Not only is the word "complex" a problem, the word "simple" is too. I mean, how "simple" are the sporadic simple groups? I have to tell my students that "simple" does not mean "not complicated", but rather "cannot be simplified further." 


I don't know of any research on this question specific to learning about mathematics. But the question opens up a big can of academic worms, outside of mathematics. In linguistics, the Whorf hypothesis (sometimes called the SapirWhorf hypothesis) can be summarized as the notion that different peoples have different languages (syntax, lexicon, etc..), and these differences influence how they think about things. For example, different languages have different tenses available for use  does this affect how speakers perceive time? So, I'd say to start by looking up the Whorf hypothesis  maybe it's been considered by some applied linguists studying education. The other linguistic can of worms is the use of metaphor in mathematical language. Some words we use are directly visual, like "smooth" and "compact", some are strange (to me) analogies like "sheaf" and "flabby", and others are part of larger metaphorical systems like "consider a variety over a finite field" (the use of the positional word "over", to express dependence like a building resting on its foundation). If you want to read up on these aspects of mathematical language, I'd recommend books by the Berkeley linguist Lakoff  the classic "Metaphors we live by", and the application to mathematics in "Where mathematics comes from". Not that I agree with everything in the latter book, but it's an interesting read. I don't think you can address your question seriously without reviewing the linguistics literature. 


This is very elementary, but I find it surprisingly common: many students talk about "infinite" numbers. You know, like $$0.3333333\ldots$$ 


Oh, and “convex”. As far as convex sets go, the mathematical usage accords well with everyday language. Not so with convex (and concave) functions. Educationists (?) have tried to remedy this by using terms “concave up” and “concave down” in calculus textbooks, a usage that I detest. (I have a hard time remembering which is which of those two.) Edited based on comments: It seems that in Russian, "convex" can refer a surface curved outward, where in English usually "convex" refers to a solid whose surface curves outward. Perhaps that's why in complex geometry, one might consider a "convex domain" or its boundary, a "convex hypersurface". 


I nominate “trivial”. It can be rather confusing that something can be trivial, but not trivially so. 


Sequence vs series? Particularly if the two notions are introduced one right after the other in a calculus course, students are doomed to mix them up. 


In italian, the words bound and limit sound the same, "limite". This often causes confusion, like in the limit points and the boundary points of a set or a bounded function and its limit. 


An example is French module monogène for what is "cyclic module" in English. I can't prove that the possibility that a module monogène for a given ring may not be a groupe cyclique would be a stumbling block for a student; but it shows the phenomenon (avoid overloading). (I learned this when Serre pulled up an anglophone lecturer speaking in French on one occasion.) 


In french, a "monic polynomial" is a "polynôme unitaire", and a "unit vector" is a "vecteur unitaire". What happens to the students when they consider a dot product on a space of polynoms? 


In french, a "unitary endomorphism" is an "endomorphisme orthogonal"... and an "orthogonal projection" is a "projection orthogonale" (the 'e' is for female, it's pronounced the same). You can hardly imagine how bad I feel when I have to tell my students that orthogonal projections aren't unitary... 


I think it takes sometime for starters to realize the tensor product between representations and tensor product between modules, despite the similarity on the surface that we can treat $g$representations as $g$modules. Usually a confusion appears when a concept which makes perfect sense in one area was redefined or used in a more subtle way in the other area, which might be counterintuitive in some sense. A concrete example comes up in my mind is Segal's paper on representations of compact Lie groups, where a lot of definitions are rather adhoc in modern literature but makes perfect sense when one read his paper careful enough. 


Parallel to most people is a precise and useful term, describing for instance railway tracks, even when curved. But when we learn "Euclid's parallel postulate", it merely means "never meeting even if extended indefinitely". This causes difficulties for students introduced to nonEuclidean geometries. 

