Perhaps this question overlaps with similar ones, ... but I want to focus on a particular possible cause of confusion. I notice that students are often confused by the concepts of "infinite" and "unbounded". So, when asked if the set of invertible matrices is compact, they reply "no, because there are an infinite number of matrices with non zero determinant, therefore the set is unbounded". Actually this happens in Italian, where the corresponding words ("infinito" and "illimitato") are almost synonyms in everyday language. Does this happen in English too, or other languages?. I wonder: what if we chose another name for the two concepts? Would they make this mistake anyway? One way to check this would be to compare with what happens in other languages, where perhaps the words chosen do not create the confusion. Do you have other examples of this situation? Can you suggest different math concepts which in one language are named with synonyms, but not in another? Do you know if this problem has been studied anywhere?
"Open" and "closed". Every reasonable human being on the planet, who has not studied topology, will assume that something can either be open or closed, but not both. This often causes students to make statements like "Set A is open, therefore it is not closed, thus ..."
The article "Surprises from mathematics education research: student (mis)use of mathematical definitions" by Edwards and Ward addresses some of your concerns in the context of U.S. undergraduates.
From the introduction:
...[T]asks involving the definitions of "limit" and "continuity," for example, were problematic for some of the students. Ward's intuitive reaction was that those words were "loaded" with connotations from their nonmathematical use and from their less than completely rigorous use in elementary calculus. He said, "I'll bet students have less difficulty or, at least, different difficulties with definitions in abstract algebra. The words there, like 'group' and 'coset,' are not so loaded."
He was surprised to see his algebra students having difficulties very similar to those of Edwards's analysis students. (So he lost his bet.) In particular, he was surprised to see difficulties arising from the students' understanding of the very nature of mathematical definitions, not just from the content of the definitions.
I don't know of any research on this question specific to learning about mathematics. But the question opens up a big can of academic worms, outside of mathematics.
In linguistics, the Whorf hypothesis (sometimes called the Sapir-Whorf hypothesis) can be summarized as the notion that different peoples have different languages (syntax, lexicon, etc..), and these differences influence how they think about things. For example, different languages have different tenses available for use -- does this affect how speakers perceive time?
So, I'd say to start by looking up the Whorf hypothesis -- maybe it's been considered by some applied linguists studying education.
The other linguistic can of worms is the use of metaphor in mathematical language. Some words we use are directly visual, like "smooth" and "compact", some are strange (to me) analogies like "sheaf" and "flabby", and others are part of larger metaphorical systems like "consider a variety over a finite field" (the use of the positional word "over", to express dependence like a building resting on its foundation).
If you want to read up on these aspects of mathematical language, I'd recommend books by the Berkeley linguist Lakoff -- the classic "Metaphors we live by", and the application to mathematics in "Where mathematics comes from". Not that I agree with everything in the latter book, but it's an interesting read. I don't think you can address your question seriously without reviewing the linguistics literature.
Oh, and “convex”. As far as convex sets go, the mathematical usage accords well with everyday language. Not so with convex (and concave) functions. Educationists (?) have tried to remedy this by using terms “concave up” and “concave down” in calculus textbooks, a usage that I detest. (I have a hard time remembering which is which of those two.)
Edited based on comments: It seems that in Russian, "convex" can refer a surface curved outward, where in English usually "convex" refers to a solid whose surface curves outward. Perhaps that's why in complex geometry, one might consider a "convex domain" or its boundary, a "convex hypersurface".
An example is French module monogène for what is "cyclic module" in English. I can't prove that the possibility that a module monogène for a given ring may not be a groupe cyclique would be a stumbling block for a student; but it shows the phenomenon (avoid overloading). (I learned this when Serre pulled up an anglophone lecturer speaking in French on one occasion.)
In french, a "unitary endomorphism" is an "endomorphisme orthogonal"... and an "orthogonal projection" is a "projection orthogonale" (the 'e' is for female, it's pronounced the same). You can hardly imagine how bad I feel when I have to tell my students that orthogonal projections aren't unitary...
I nominate the word "kernel" as the most abused term in mathematics. The null space of a linear operator, a symmetric positive definite function on a set, a smooth probability density peaked -- am I missing any? I know these are all vaguely related, but the polysemy definitely causes much confusion for students.
I think it takes sometime for starters to realize the tensor product between representations and tensor product between modules, despite the similarity on the surface that we can treat $g$-representations as $g$-modules. Usually a confusion appears when a concept which makes perfect sense in one area was re-defined or used in a more subtle way in the other area, which might be counter-intuitive in some sense. A concrete example comes up in my mind is Segal's paper on representations of compact Lie groups, where a lot of definitions are rather ad-hoc in modern literature but makes perfect sense when one read his paper careful enough.
Parallel to most people is a precise and useful term, describing for instance railway tracks, even when curved. But when we learn "Euclid's parallel postulate", it merely means "never meeting even if extended indefinitely". This causes difficulties for students introduced to non-Euclidean geometries.