With which notation do you feel uncomfortable?

22$\begingroup$ I'm just crotchety, and don't like this type of question. $\endgroup$– Theo JohnsonFreydCommented Mar 18, 2010 at 16:18

1$\begingroup$ At while back I was considering asking a similar question, but people quickly convinced me that it was a bad idea (tea.mathoverflow.net/discussion/100/…). The basic argument is that this question is about as subjective and argumentative as math gets. $\endgroup$– Anton GeraschenkoCommented Mar 18, 2010 at 17:38

1$\begingroup$ I don't know, I like some of the answers. Although, it is in danger of a small signaltonoise ratio. $\endgroup$– Ilya GrigorievCommented Mar 18, 2010 at 17:54

5$\begingroup$ Closed. I know, I know, it's nice to have some questions like this to waste some time on, but we really don't need to have this at the top of the home page for several days. I'd recommend going to vote up this feature request meta.stackexchange.com/questions/3414/… if you happen to have meta.SE rep, and feel the same way. $\endgroup$– Kim MorrisonCommented Mar 19, 2010 at 6:13
22 Answers
There is a famous anecdote about Barry Mazur coming up with the worst notation possible at a seminar talk in order to annoy Serge Lang. Mazur defined $\Xi$ to be a complex number and considered the quotient of the conjugate of $\Xi$ and $\Xi$: $$\frac{\overline{\Xi}}{\Xi}.$$ This looks even better on a blackboard since $\Xi$ is drawn as three horizonal lines.

19$\begingroup$ The full story involves a tshirt that Barry had commissioned and worn for the occasion... $\endgroup$ Commented Mar 18, 2010 at 15:44

10$\begingroup$ @Pete: Can you fill us in with the full story? $\endgroup$ Commented Mar 18, 2010 at 19:20

32$\begingroup$ I should mention that I am a Mazur student, hence a designated keeper of the flame of Mazur lore (although there are about 50 other such people) but that I was not there to see the event myself. (Actually I never met Lang, to my regret.) The most authoritative description I have found is given by Paul Vojta here: ams.org/notices/200605/fealang.pdf $\endgroup$ Commented Mar 18, 2010 at 21:06

12$\begingroup$ I had heard two additional versions of this story that differed from Vojta's in the ending, in the sense that Vojta's Serge didn't make a peep, and the other two Serges exploded. $\endgroup$– S. Carnahan ♦Commented Mar 19, 2010 at 5:06

17$\begingroup$ @PeteL.Clark: About a year ago I was at a reception with Mazur and Rohrlich, and asked Mazur what had really happened during that lecture. What he described matched Vojta's description, except that Lang did not keep quiet. Instead, he said (calmly) to Mazur "You really seem to have a thing for horizontal lines". Since the intended reaction had not been produced, Mazur then opened the box containing the shirt and gave it to Serge. $\endgroup$– KConradCommented Jul 14, 2014 at 4:08
My favorite example of bad notation is using $\textrm{sin}^2(x)$ for $(\textrm{sin}(x))^2$ and $\textrm{sin}^{1}(x)$ for $\textrm{arcsin}(x)$, since this is basically the same notation used for two different things ($\textrm{sin}^2(x)$ should mean $\textrm{sin}(\textrm{sin}(x))$ if $\textrm{sin}^{1}(x)$ means $\textrm{arcsin}(x)$).
It might not be horrible, since it rarely leads to confusion, but it is inconsistent notation, which should be avoided in general.

67$\begingroup$ I take issue with "it rarely leads to confusion". Among mathematicians that's true, but among calculus students is another story. $\endgroup$ Commented Mar 18, 2010 at 18:54

19$\begingroup$ I refuse to use the notation $\sin^{1}(x)$, except perhaps to complain about it, and use the notation $\arcsin(x)$. This means that anything I do with nontrivial powers or inverses of trig functions use a different convention to the rest of any work I do with functions, but it is better than the ugliness of a notation with a single function which is inconsistent. $\endgroup$ Commented Mar 18, 2010 at 22:24

41$\begingroup$ In my first semester of college I was marked wrong on an exam for writing $\sin^{1}(x)$ on a question whose answer was $\arcsin(x)$. When I complained, the professor kept explaining why $1/\sin(x)$ was incorrect. He claimed never to have heard of this notation for arcsin, and relented only after I got a second opinion from another professor in the department. So it's not only calculus students who get confused by this! $\endgroup$ Commented Mar 19, 2010 at 1:40

15$\begingroup$ Actually, I take issue with both of these notations. :) As a dynamicist, I would indeed agree that $\sin^2$ should be the second iterate of sine. However, $\sin$ is not invertible, and hence $\sin^{1}$ should not be used for the arcsine, which is only a specific branch of the inverse function. $(\sin_{[\pi/2,\pi/2]})^{1}$ would be ok I suppose ... $\endgroup$ Commented Jan 26, 2015 at 16:29

11$\begingroup$ Why not writing $f^{\circ 2} = f \circ f$ and $f^{\circ 1}$? And $f^2$, $f^{1}$ are defined pointwise as usual. $\endgroup$ Commented Sep 13, 2015 at 8:25
I personally hate the notation $x \mid y$, for "$x$ divides $y$". Of course, I'm used to reading it by now, but a general principle I follow and recommend is:
Never use a symmetric symbol to denote an asymmetric relation!

55$\begingroup$ How do you cope with wedge products? Or tensor products? or, gasp, multiplication in a noncommutative group? I hope I didn't just blow your mind. :p $\endgroup$ Commented Mar 18, 2010 at 16:55

21$\begingroup$ Graham, Knuth, and Patashnik, in "Concrete Mathematics", share your unease. They write x\y for "x divides y". $\endgroup$ Commented Mar 18, 2010 at 17:08

57$\begingroup$ Regarding wedge, tensor, multiplication symbols  I specifically said "relation", not "operation". And I stand by what I said  I cope just fine! :) $\endgroup$– MartyCommented Mar 18, 2010 at 17:55

22$\begingroup$ Yes, when I tell undergraduates that x divides y means y/x is an integer I can see them thinking, "No way, he's messing with us." Then I tell them that actually 0 divides 0 as well, and I can practically see the steam coming out of their ears. (Come to think of it, this gives a funny variation on the old precalculus brainteaser "What is zero over zero"? Answer: "Yes.") $\endgroup$ Commented Mar 18, 2010 at 21:15

33$\begingroup$ I also don't like the notation $x \backslash y$ for "x divides y". That seems likely to confuse students due to its resemblance to an operation. If I were to choose a symbol, I might use $x \triangleleft y$, to mean "x divides y". It preserves the vertical line, which we're all used to, and adds a directional component which I find appropriate. $\endgroup$– MartyCommented Mar 18, 2010 at 22:18
Mathematicians are really quite bad when it comes to notation. They should learn from programming languages people. Bad notation actually makes it difficult for students to understand the concepts. Here are some really bad ones:
 Using $f(x)$ to denote both the value of $f$ at $x$ and the function $f$ itself. Because of this students in programming classes cannot tell the difference beween $f$ (the function) and $f(x)$ (function applied to an argument).
 When I was a student nobody ever managed to explain to me why $dy/dx$ made sense. What is $dy$ and what is $dx$? They're not numbers, yet we divide them (I am just giving a student's perspective).
 In Langrangian mechanics and calculus of variations people take the partial derivative of the Lagrangian $L$ with respect to $\dot q$, where $\dot q$ itself is the derivative of momentum $q$ with respect to time. That's crazy.
 The summation convention, e.g., that ${\Gamma^{ij}}_j$ actually means $\sum_j {\Gamma^{ij}}_j$ is useful but very hard to get used to.
 In category theory I wish people sometimes used any notation as opposed to nameless arrows which are introduced in accompanying text as "the evident arrow".

44$\begingroup$ Your first point is not a problem with the notation but with the users of the notation... $\endgroup$ Commented Mar 18, 2010 at 15:23

100$\begingroup$ The Leibniz notation dy/dx for the derivative is perhaps the most clever bit of notation ever invented in the whole history of mathematics. $\endgroup$ Commented Mar 18, 2010 at 15:51

45$\begingroup$ Einstein summation is indeed dumb, not because it is hard to get used to, but because it makes it impossible to refer to a single addend rather than to the whole sum (as in: "each of the addends is nonnegative, hence the sum is nonnegative"), forcing writers to leave out parts of their arguments just because they can't write it with Einstein summation. Although someone has proposed a sum sign crossed out to mean "this is NOT a sum". $\endgroup$ Commented Mar 18, 2010 at 15:58

50$\begingroup$ The Einstein summation convention is beautiful. It essentially makes it impossible to write down something that isn't coordinate independent. Or rather it's Penrose's abstract index notation that does this. But as his notation is essentially identical to Einstein's, there's not much difference. en.wikipedia.org/wiki/Abstract_index_notation (@andrej I thought you'd appreciate that kind of "type safety".) (@darij The fact that you can't refer to individual components is a virtue. It encourages people to state and write proofs that are independent of choice of basis.) $\endgroup$ Commented Mar 18, 2010 at 16:14

35$\begingroup$ In my first calculus class, a very big deal was made to make sure students understood that dy/dx was NOT a quotient; dy and dx had NO meaning whatsoever on their own, it was just notation. And then I got to differential equations, and on the first day the professor said "Now multiply by dx." The other students seemed perfectly OK with this, but it confused the heck out of me for a while. Maybe I was the only one who had actually believed the calculus professor that dx had no independent meaning. $\endgroup$ Commented Mar 18, 2010 at 19:45
I never liked the notation ${\mathbb Z}_p$ for the ring of residue classes modulo $p$. At one point, it confused the hell out of me, and this confusion is easily avoided by writing $C_p$, $C(p)$ or ${\mathbb Z}/p$.

17$\begingroup$ Yep. $\mathbb{Z}_p$ is the ring of $p$adic integers. But it took me some time to learn that too... $\endgroup$ Commented Mar 18, 2010 at 16:06

28$\begingroup$ There is a slight ambiguity here. My personal convention is that C_p and C(p) refer to the cyclic groups with no mention of their ring structure, and it is Z/pZ (or even better, F_p) which refers to the actual ring. $\endgroup$ Commented Mar 18, 2010 at 17:01

25$\begingroup$ Please don't confuse the residueclass ring ${\mathbb Z}_q$ (aka ${\mathbb Z}/q{\mathbb Z}$) with a Galois field of $q$ elements, conveniently denoted by ${\mathbb F}_q$ (or $GF(q)$). It is one of the most popular fallacies of our students to assume that both symbols denote the same mathematical object, even if $q$ is not a prime. $\endgroup$– MRACommented Mar 18, 2010 at 17:01

17$\begingroup$ You can never go wrong with $\mathbb{Z}/p\mathbb{Z}$ or $\mathbb{Z}/\mathfrak{p}$. $\endgroup$ Commented Mar 18, 2010 at 18:46

31$\begingroup$ Jim, I disagree. It makes sense if we agree that it should make sense. It's unambiguous. And it follows a noble tradition of ignoring the distinction between elements of a ring and the principal ideals that they generate. $\endgroup$ Commented Jan 8, 2012 at 0:56
Physicist will hate me for this, but I never liked Einstein's summation convention, nor the famous bra ($\langle\phi$) and ket ($\psi\rangle$) notation. Both notations make easy things look unnecessarily complicated, and especially the braket notation is no fun to use in LaTeX.

294$\begingroup$ Personally, I strongly dislike the misuse of the relation symbols
$<$
and$>$
instead of the appropriate$\langle$
and$\rangle$
angle brackets. I dislike it so strongly, in fact, that I edited this answer. $\endgroup$ Commented Mar 18, 2010 at 19:16 
31$\begingroup$ I would vote up the previous comment seven times if I could. $\endgroup$ Commented Mar 18, 2010 at 19:50

15$\begingroup$ You can use latex's newcommand to make the typesetting easier. Type \newcommand{\bra}[1]{\langle #1} at the top of your document, and \bra\psi will produce $\newcommand{\bra}[1]{\langle #1} \bra\psi$ $\endgroup$– user1504Commented Mar 18, 2010 at 20:10

24$\begingroup$ Dirac notation isn't really justifiable for generic vectors on their own. But it is so helpful for quantum information! It is so much easier to read/write standard basis vectors as $a_1 a_2 \cdots a_n\rangle$ than as $\mathbf{e}_{a_1,a_2,\ldots,a_n}$ (or worse: $\mathbf{e}_{a_1} \otimes \cdots \otimes \mathbf{e}_{a_n}$). I would go further, and introduce this notation into introductions to probability. For expressing states of configuration spaces over distinguishable labels, it is quite good. $\endgroup$ Commented Mar 18, 2010 at 22:34

19$\begingroup$ I would say the Einstein summation notation makes complicated things look misleadingly easy. When I write that scalar curvature is $R=g^{ij}R^k_{ikj}$, is it clear how much is actually going on there? It does prevent you from writing anything coordinatedependent, which is quite nice. $\endgroup$ Commented Mar 19, 2010 at 1:53
I think composition of arrows $f:X\to Y$ and $g:Y\to Z$ should be written $fg$ not $gf$. First of all it would make the notation $\hom(X,Y)\to\hom(Y,Z)\to \hom(X,Z)$ much more natural: $\hom(E,X)$ should be a left $\hom(E,E)$ module because $E$ is on the left :) Secondly, diagrams are written from left to right (even stronger: Almost anything in the western world is written left to right). And i think the strange (1) needed when shifting complexes is an effect of this twisted notation.

12$\begingroup$ To some extent I agree. The problem is that the order in which composition is written in standard notation is the opposite of the temporal order, which by any standard is the "natural" one  first do this, then do this, then do this. In fact, I think it's reasonable to argue that the basis of the standard definition of function is the arrow of time. $\endgroup$ Commented Mar 18, 2010 at 16:58

7$\begingroup$ Harald it was Herstein, the author of our favourite algebra book! $\endgroup$– MaharanaCommented Mar 18, 2010 at 17:41

7$\begingroup$ Jacobson's book on Lie algebras also uses $xf$ to mean $f(x)$. It seems that at the time this notation was all the rage. I find it very confusing now. $\endgroup$ Commented Mar 18, 2010 at 18:34

25$\begingroup$ $xf$ makes a lot of sense when you realize that $x: 1 \rightarrow X$ and $f: X \rightarrow Y$ are both functions. $\endgroup$ Commented Mar 19, 2010 at 13:55

9$\begingroup$ I believe the primary source of confusion here is that there is no reason whatsoever for the arrows to go from left to right. If by chance it would begin with something like $f:Y\leftarrow X$... $\endgroup$ Commented Jun 7, 2015 at 5:17
The notation ]a,b[ for open intervals and its ilk. Sorry, Bourbaki.

49$\begingroup$ I like ]a,b[ because it does not have the ambiguity of (a,b) and it's very clear that a,b are excluded. $\endgroup$ Commented Apr 16, 2010 at 15:07

32$\begingroup$ Personally, i have nothing against $]a,b[$; it just makes the `parenthesis (bracket) matcher' of my text editor blush. $\endgroup$– SuvritCommented Oct 17, 2010 at 8:47

43$\begingroup$ Imagining $a$ and $b$ on a real line, the notation $]a,b[$ looks to me more like $\mathbb R \setminus (a,b)$... $\endgroup$– Y. PeiCommented Feb 5, 2015 at 18:48

26$\begingroup$ Find all $(a,b)$ such that $(a,b)\in(a,b)$ $\endgroup$ Commented Oct 14, 2015 at 0:23

24$\begingroup$ @LSpice: The first $(a,b)$ is an ordered pair, the second $(a,b)$ is the gcd, the third $(a,b)$ is an interval. I leave it to you to solve the exercise. $\endgroup$ Commented Nov 3, 2015 at 17:07
My candidate would be the (internal) direct sum of subspaces $U \oplus V$ in linear algebra. As an operator it is equivalent to sum but with the side effect of implying that $U \cap V = \lbrace 0\rbrace$. Whenever I had a chance to teach linear algebra I found this terribly confusing for students.

8$\begingroup$ Shouldn't it imply
$U \cap V = \{0\}$
? I guess that's another piece of bad notation: not all trivial things equal $\varnothing$. $\endgroup$ Commented Mar 19, 2010 at 0:10 
1$\begingroup$ @Francois: Sheesh, of course. Sorry. Fixed. $\endgroup$ Commented Mar 19, 2010 at 0:20

1$\begingroup$ This isn't confusing as long as you clearly distinguish interior and exterior direct sums. $\endgroup$ Commented Mar 19, 2010 at 5:16

6$\begingroup$ @Qiaochu: I'm really only talking about the interior case. It's then that the \osum is the same subspace as the sum but with the implied additional condition on the subspaces. $\endgroup$ Commented Mar 19, 2010 at 7:16

1$\begingroup$ What would you do instead, write $U + V$ and note that $U \cap V = \{0\}$ on the side? I imagine that you might have to be more explicit (at least in introductory material) with arguments of the form ‹Because $U \cap V = \{0\}$, $U + V \cong U \oplus V$, so ….›. (I'm not claiming that this is a bad thing!) $\endgroup$ Commented Feb 27, 2017 at 14:43
Writing a finite field of size $q$ as $\mathrm{GF}(q)$ instead of as $\mathbf{F}_q$ always rubbed me the wrong way. I know where it comes from (Galois Field), and I think it is still widely used in computer science, and maybe in some allied areas of discrete math, but I still dislike it.

11$\begingroup$ Well, maybe $\mathrm{GF}(q^{q^2})$ is still better than $\mathbb F_{q^{q^2}}$. Indices are nice until one has to nest three or more of them... $\endgroup$ Commented Mar 18, 2010 at 20:10

6$\begingroup$ In references authors may write exp($z$) instead of $e^z$ when the exponent could get particularly complicated, but when does this ever really happen for sizes of finite fields? If I find it necessary to write something like ${\mathbf F}_{q^{q^2}}$ again, I'll reconsider my opposition to writing GF. :) $\endgroup$– KConradCommented Mar 18, 2010 at 21:20

5$\begingroup$ Best thing is when it's (re)translated as "GaloisFeld" in German. $\endgroup$– user717Commented Mar 18, 2010 at 23:36

3$\begingroup$ The worst thing about "GaloisFeld" is that it has apparently been the standard notion in German in the beginning of the 20th Century (at least, Witt uses it).  @KConrad: "Contributions to the Theory of Finite Fields" by Oystein Ore (Transactions of the American Mathematical Society, Vol. 36, No. 2. (Apr., 1934), pp. 243274) works with $\mathbb{F}_{p^{ff^{\prime}}}$, which he fortunately abbreviates. Otherwise, I think not even JSTOR's high scanning quality would suffice to recognize the symbols. $\endgroup$ Commented Mar 18, 2010 at 23:54

1$\begingroup$ @darij: Are you sure? "Field" is "Körper" in German and always has been. I don't know why it's called "field" in English, but it's okay. Only when people then translate "Galois field" as "GaloisFeld", ... :/ $\endgroup$– user717Commented Mar 19, 2010 at 12:03
I rather dislike the notation $$\int_{\Omega}f(x)\,\mu(dx)$$ myself. I realize that just as the integral sign is a generalized summation sign, the $dx$ in $\mu(dx)$ would stand for some small measurable set of which you take the measure, but it still rubs me the wrong way. Is it only because I was brought up with the $\int\cdots\,d\mu(x)$ notation? The latter nicely generalizes the notation for the Stieltjes integral at least.

6$\begingroup$ I also used to dislike that notation, but I think it's only a matter of upbringing. It's useful, e.g., for dealing with things like Markov kernels: when you have $K:\Omega\times\mathcal{F}\to\mathbb{R}$, where $\mathcal{F}$ is a $\sigma$algebra on $\Omega$ and for each $x$, $K(x,\cdot)$ is a measure, you can easily write things like $\int f(x)K(y,dx)$, which is much harder to write with the $\int f(x) d\mu(x)$ notation. $\endgroup$ Commented Mar 18, 2010 at 19:59

2$\begingroup$ Also, whether $\int f(x) d\mu(x)$ nicely generalizes the Stieltjes integral notation or not depends on your viewpoint. If $\frac{d\mu}{dx} = h = \frac{d H}{dx}$, then indeed $\int f(x) d\mu(x) = \int f(x) dH(x)$. On the other hand, in that case $\mu=h$ (not $H$) as distributions, i.e. $\langle f,\mu \rangle = \langle f,h \rangle$. $\endgroup$ Commented Mar 18, 2010 at 20:05

4$\begingroup$ That is one of the worse. Formally you have to write $(f\mu)(\Omega)$, but no one does it. It should be form under the integral, so in principle you can think that measure is a form so you can write $\int f\mu$ and no one does it either... Instead everyone writes something which has no sense like $\int f\mu(dx)$ or $\int fd\mu$... – Anton Petrunin 0 secs ago $\endgroup$ Commented Mar 20, 2010 at 16:44

3$\begingroup$ The $d$ is entirely superfluous in $\int \cdots d\mu(x)$ (which does not generalize the Stieltjes integral, because the measure in $\int \cdots dg(x)$ is $dg$, not $g$). You should either write $\int \cdots \mu(dx)$ or (if you're willing to defy convention) $\int \cdots \mu(x)$ (or $\int \cdots \mu$ if there is no need to use a dummy variable). You can see this in action at ncatlab.org/nlab/show/measure+space. $\endgroup$ Commented Feb 27, 2017 at 15:07
I get very frustrated when an author or speaker writes "Let $X\colon= A\sqcup B$..." to mean:
 $A$ and $B$ are disjoint sets (in whatever the appropriate universe is),
 and let $X\colon= A\cup B$.
If they just meant "form the disjoint union of $A$ and $B$" this would be fine. But I've seen speakers later use the fact that $A$ and $B$ are disjoint, which was never stated anywhere except as above. You should never hide an assumption implicitly in your notation.

4$\begingroup$ But the point is that there is no universally agreed upon definition of what the "disjoint union" of two nondisjoint sets is. Or is there? (I suppose if you forced me to give one, I would put
$A \coprod B = A \times \{0\} \cup B \times \{1\}$
, but surely not everyone agrees with this?) $\endgroup$ Commented Mar 19, 2010 at 6:18 
13$\begingroup$ This is very similar to Alon Amit's post about writing internal direct sums using the more general direct sum notation without mention of the implicit assumptions. The problem is when the notation does not refer to the "disjoint union" of notnecessarilydisjoint sets, but rather to a union of two sets that are assumed implicitly to be disjoint. $\endgroup$ Commented Mar 19, 2010 at 9:28

39$\begingroup$ Oh for Pete's sake! :D Seriously, Pete: what's the problem? The disjoint union is defined up to unique isomorphism by a universal property; the particular settheoretic encoding is of no importance whatever. $\endgroup$– Todd Trimble ♦Commented Nov 19, 2012 at 2:41

3$\begingroup$ I agree with Tom Church that the cited usage is inept. $\endgroup$– Todd Trimble ♦Commented Nov 19, 2012 at 2:44

3$\begingroup$ I agree with Todd Trimble. Also, notice that there is no notion of "two sets are disjoint" for a category theorist. It only makes sense to say that two arrows $A \to X \leftarrow B$ are disjoint. $\endgroup$ Commented Sep 13, 2015 at 8:33
As Trevor Wooley used to always say in class, “Vinogradov's notation sucks…the constants away.”
For those who don't know, Vinogradov's notation in this context is $f(x)\ll g(x)$ meaning $f(x) = O(g(x))$. (If you prefer bigO notation, that is.)

19

12$\begingroup$ For the Big O notation "sucks" and Vinogradov's notation has sense and. Say $o(x)=O(x)$ but $O(x)\not=o(x)$. $\endgroup$ Commented Mar 18, 2010 at 23:40

16$\begingroup$ The above is not a problem caused by bigO notation, but a problem caused by failing to recognize that $O(x)$ is a set. The correct equivalent statements are that $o(x) \subseteq O(x)$ but $O(x) \not\subseteq o(x)$. [Of course, $o(x) \ne O(x)$ is also correct.] As equality is the most important relation we have, one should never write an equals sign unless one really means it! $\endgroup$ Commented Mar 19, 2010 at 6:24

29$\begingroup$ So $f(x) < \!\! < f(x)$? That's terrible. Weak inequalities should have a horizontal line in their notations, like $\leq$ and $\subseteq$. $\endgroup$ Commented Mar 19, 2010 at 7:12

8$\begingroup$ @LSpice I've actually taken to writing $\subseteq$ or $\subsetneq$ whenever the issue is important and not obvious from context. While I agree that $\subset$ should mean $\subsetneq$, there is not enough consensus on this point to be sure I am understood. $\endgroup$ Commented Oct 13, 2015 at 20:40
The use of squared brackets $\left[...\right]$ for anything. It's not bad per se, but unfortunately it is used both as a substitute for $\left(...\right)$ and as a notation for the floor function. And there are cases when it takes a while to figure out which of these is meant  I'm not making this up.
The word "character" meaning: a 1dimensional representation, a representation, a trace form of a representation, a formal linear combination of representations, a formal linear combination of trace forms of representations.
The word "adjoint", and the corresponding notation $A\mapsto A^{\ast}$, having two completely unrelated meanings.

4$\begingroup$ Do people really use squared brackets for the floor function? I thought it has its own symbol (one that makes a lot of sense at that). Also, if you want to talk about words that are abused, at least be outraged by "normal"! $\endgroup$ Commented Mar 18, 2010 at 16:16

26$\begingroup$ Indeed, the floor function should be written
$\lfloor\cdot\rfloor$
. Pet peeve of mine. $\endgroup$ Commented Mar 18, 2010 at 16:24 
2$\begingroup$ Yes, squared brackets are still used in some parts of the world for floor, unfortunately. And as for "normal", it indeed belongs into the list, though it's not as bad as people claim; the different uses of "normal" mostly belong to different fields of mathematics, and thus it's not that easy to confuse them. Except "normal" for Hopf algebras vs. "normal" for commutative rings. $\endgroup$ Commented Mar 18, 2010 at 16:33

4$\begingroup$ @darij: Take a topological group $G$, now consider a normal subgroup $H$ ... $\endgroup$ Commented Mar 18, 2010 at 18:54

1$\begingroup$ Wasn't there a topology textbook that had two incompatible definitions of "perfectly normal"? (I think it was a translation and the original used different words for "perfect", but still...) $\endgroup$ Commented Mar 19, 2010 at 0:31
The term "symplectic group" used to mean the group $U(n,{\mathbb H})$. It's as if people called $U(n)$ and $GL(n,{\mathbb R})$ by some single name.

2$\begingroup$ Allen, that's amazing, and I'd never heard of it! Where did it happen? $\endgroup$– LSpiceCommented Sep 8, 2010 at 4:05

$\begingroup$ Re, answer to myself: Weyl, probably? $\endgroup$– LSpiceCommented Aug 21, 2023 at 22:59
My personal pet peeve of notation HAS to be algebraists writing functions on the right a la Herstein's "Topics In Algebra". I don't know why they do it when everyone else doesn't. I think one of them got up one day and decided they wanted to be cooler then everyone else, seriously...

38$\begingroup$ The rule isn't strange, but it is an example of “rationalization” efforts going too far. Like having 100 degrees (or “grads”) in a right angle, or my favourite, the 13 month calendar. The idea is quite clever, actually: 13 months, each 28 days long, add up to 364 days. The 365th day (and the 366th, in leap years) should be a universal holiday, and it should not be assigned a weekday name! Thus the calendar for every month would look the same – from Monday the 1st to Sunday the 28th. But thinking you can actually inflict this on society verges on lunacy. $\endgroup$ Commented Mar 18, 2010 at 19:29

51$\begingroup$ Come on, don't tell me that the "lunacy" pun in the context of the 13 months calendar was unintentional... $\endgroup$ Commented Mar 18, 2010 at 20:11

19$\begingroup$ The "functions on the right" style is alive and well... in many object oriented programming languages. Methods are functions applied to objects. $\endgroup$ Commented Mar 18, 2010 at 22:07

5$\begingroup$ Some genius among the programmers of GAP must have deemed it a great idea to drag that unholy right function notation out of its welldeserved grave and force it upon users. To make things worse, $f(x)$ is written x^f in GAP, which means additional fun because of the way systems process the tilde sign. And, of course, it is a consequence that group multiplication on $S_n$ in GAP is opposite to the rest of the world. Dear Mr. Cool, thanks for proving once again that open source software is not written for users. $\endgroup$ Commented Sep 12, 2010 at 23:34

13$\begingroup$ I once had the pleasure of teaching a course ("applied modern algebra") from a textbook that used the "composition on the right" convention for binary relations, defined functions as a special case of binary relations but used composition on the left for them, and defined permutations as a special case of functions but used composition on the right for them. $\endgroup$ Commented Mar 30, 2011 at 16:22
I don't like (but maybe for a bad reason) the notation $F\vdash G$ for $F$ is left adjoint to $G$.
Any comment ?

2$\begingroup$ If we ignore the extra information (unit and counit), then adjointness is a binary relation. Presumably then you want to use the notation which places a symbol between F and G. What is wrong with $\dashv$? Would you prefer another symbol? $\endgroup$ Commented Mar 18, 2010 at 15:24

31$\begingroup$ Isn't it $F\dashv G$ for "F left adj. to G"? $\endgroup$ Commented Mar 18, 2010 at 15:33

10$\begingroup$ I think Garief's question suggests why this is bad notation: I doubt I could ever remember which way it is supposed to go without checking wikipedia. Either convention would make sense since F is on the left. Using a symmetric symbol here is definitely a bad idea, but when I need to abbreviate "F is left adjoint to G" I often write something like "F:D <> C:G is an adjunction". $\endgroup$ Commented Mar 18, 2010 at 17:09

6$\begingroup$ The only way I can remember which one is which is by looking at $Hom(F(\cdot),\cdot) \cong Hom( \cdot ,G(\cdot))$ $\endgroup$ Commented Mar 18, 2010 at 18:49

6$\begingroup$ I recently realized that the mnemonic for the unit and counit is that the left (resp. right) adjoint occurs visually first when it occurs on the left (resp. right) side of the arrow. So if F is left adjoint to G then the unit and counit are 1 > GF and FG > 1. $\endgroup$ Commented Mar 18, 2010 at 23:53
A cute idea but for which I have yet to find supporters is D. G. Northcott's notation (used at least in [Northcott, D. G. A first course of homological algebra. Cambridge University Press, London, 1973. xi+206 pp. MR0323867) for maps in a commutative diagram, which consists in enumerating the names of the objects placed vertices along the way of the composition. Thus, if there is only one map in sight from $M$ to $N$, he writes it simply $MN$, so he has formulas looking like $$A'A(ABB'') = A'ABB'' = A'B'BB'' = 0.$$ He also writes maps on the right, so his $$xMN=0$$ means that the image of $x$ under the map from $M$ to $N$ is zero.
I would not say this is among the worst notations ever, though.
Students have big difficulties when first confronted with the $o(\cdot)$ and $O(\cdot)$ notation. The term $o(x^3)$, e.g., does not denote a certain function evaluated at $x^3$, but a function of $x$, defined by the context, that converges to zero when divided by $x^3$.

6$\begingroup$ And for the littleoh notation in particular, one often forgets to specify the limit. There is a rather substantial difference between “
$o(x^3)$
as$x\to\infty$
” and “$o(x^3)$
as$x\to0$
”, after all. $\endgroup$ Commented Mar 18, 2010 at 19:36 
4$\begingroup$ @exfalsoquodlibet: How would you prefer to express these concepts? It is possible to express these concepts quite nicely as limits of ratios, but the bigO, littleo notation has the advantage of being fairly clear while expending less effort  which is ultimately the goal of notational devices. $\endgroup$ Commented Mar 18, 2010 at 22:16

9$\begingroup$ exfalsoquodlibet Big O notation is useful because I can include it as a term in larger expressions. For example,
$$\sum \log(1/(1\chi(p)/p^s)) = \sum \left( \chi(p)/p^s + O(1/p^{2s}) \right) = \sum \chi(p)/p^s + O(1)$$
as $s \to 1^{+}$. Try writing that in your preferred notation and see which is more readable. $\endgroup$ Commented Mar 19, 2010 at 7:19 
3$\begingroup$ I find the only thing wrong with the big O notation is the equality symbol: $f = O(g)$. It's really a reflexive transitive relation, so one should write $f \leq_O g$ or something. $\endgroup$– Todd Trimble ♦Commented Feb 26, 2013 at 1:35

4$\begingroup$ @StevenStadnicki Very belated response, but: yes, that would be a very proper use of notation. In fact there's a notation pun that I like: if we think of Hardy fields that consist of germs of functions at infinity as valuation fields $K$, and remember that in algebraic geometry one often uses $O$ or $\mathcal{O}$ for the local valuation ring of germs that are bounded at the point (in this case $\infty$), then your $f \in O(g)$ literally means $f \in O g$ in the sense of principal fractional ideals (the same as $f \leq g$ in the ordered valuation group $K^\ast/O^\ast$). $\endgroup$– Todd Trimble ♦Commented Oct 13, 2015 at 22:15
I have struggled with 'dx'. I've spent years trying to study every different approach to calculus that I could find to try and make sense of it. I read about the limit definitions in my first book, vector calculus with them as pullbacks of linear transformations or flows/flux, differential forms from the bridge project, kforms, nonstandard analysis which enlarges $\mathbb{R}$ to give you infinitesimals (and unbounded numbers) but the same first order properties and lets integral be defined as a sum, constructive analysis using a monad to take the closure of the rationals to give reals... but I am still just as confused as ever, I understand that the mathematical notation doesn't have a compositional semantics but still don't really get it  one of the problems is despite not really understanding it, or having any abstract definition of it.. I can still get correct answers and I really hope this doesn't become a theme as I study more topics in mathematics.

4$\begingroup$ Almost always, dx refers to the 1form obtained by exterior differentiation from the function x on your space (say the graph of a function R>R viewed as a subset of R^2, in which case x is the projection onto the first factor). It's not really mysterious. I don't know about nonstandard analysis, but I can imagine it would mean something else in such contexts. Although in my opinion, authors would do well to use a new symbol to denote something like "infinitessimal change in x" (assuming they can make mathematical sense of this) rather than overloading a symbol with a perfectly good meaning $\endgroup$ Commented Mar 18, 2010 at 17:18

2$\begingroup$ I'm voted 1 for this remark? Is it because it's too basic or boring a problem or why? :/ $\endgroup$– muadCommented Mar 18, 2010 at 18:22

6$\begingroup$ Probably you were downvoted because you did not really explicitely explain the confusion caused by dx. As somebody who uses differential forms a lot I have never stumbled over a situation where dx was mysterious for me. $\endgroup$ Commented Mar 18, 2010 at 18:56

6$\begingroup$ There are definitely some annoying things about the $dx$ or $\frac d{dx}$ notation, but I think Newton's notation was worse, so $dx$ can't be the worst. $\endgroup$ Commented Mar 18, 2010 at 18:56

22$\begingroup$ Then how about $dS=\sqrt{dx^2+dy^2}$ (<some physicists love this) ?!... :) $\endgroup$– M.G.Commented Mar 18, 2010 at 20:26
p < q as in "the forcing condition p is stronger than q".

1$\begingroup$ Think of intervals: A smaller interval contains more information in the sense that it determines an arbitrary element with less error. Hence giving the smaller interval is the "stronger" condition. (If I remember correctly, there really is a forcing notion where this is literally true. The poset it choosen to be the Borel sets of [0,1] or something similar. Please correct me if that's wrong, I've never really had anything to do with forcing) $\endgroup$ Commented Dec 1, 2013 at 1:02

1$\begingroup$ I don't think it is an issue, since this is the appropriate ordering in the Boolean completion, where the weakest condition, 1, is the largest. Anyway, Matt Foreman avoids having to decide whether to use this or the other convention by writing $p\Vdash q $ and only using separative posets. $\endgroup$ Commented Jul 12, 2017 at 23:15
I hate the short cut $ab$ for $a\cdot b$. Everyone get used to it, BUT it creates very deep problem with all other notation; say you never can be sure what $f(x+y)$ or $2\!\tfrac23$ might be...
Also in modern mathematics people do not multiply things too often, so it does not have sense to make such a short cut.
Yet the shortcut $x^n$ is really bad one. One can not use upper indexes after this. It would be easy to write $x^{\cdot n}$ instead.

10$\begingroup$ Writing $a\dot b$ every time would make group presentation theorems a night mare to read. $\endgroup$ Commented Mar 18, 2010 at 21:01

6$\begingroup$ @Anonymous. No, it would not. $\endgroup$ Commented Mar 18, 2010 at 21:35

15$\begingroup$ At the risk of stating the obvious, I will venture that this convention exists for a simple reason. In a tremendous number of situations, it makes parsing a mathematical expression roughly equivalent in effort to parsing a written sentence in a European language; thus reducing an important task to one which had been previously solved. $\endgroup$ Commented Mar 18, 2010 at 22:40

11$\begingroup$ Can you elaborate what deep problems are created by this shortcut? $\endgroup$ Commented Mar 19, 2010 at 0:01

8$\begingroup$ BTW, I am impressed by number of negative votes  I think it only shows that we are not ready to admit that the notation we use everyday is the worst one. $\endgroup$ Commented Dec 1, 2013 at 20:14