With which notation do you feel uncomfortable?

8$\begingroup$ This is, at best, a community wiki question... $\endgroup$ – Mariano SuárezÁlvarez Mar 18 '10 at 14:58

6$\begingroup$ Mariano, it looks like you started this question. I had to check the edit history to confirm that it wasn't you =(. I also misread your comment, which I thought said "This is the best community wiki question." $\endgroup$ – Harry Gindi Mar 18 '10 at 15:13

19$\begingroup$ I'm just crotchety, and don't like this type of question. $\endgroup$ – Theo JohnsonFreyd Mar 18 '10 at 16:18

3$\begingroup$ @Theo, me too. I would encourage voting to close on biglist questions that you feel are subjective. If my vote weren't binding (or there were already 3 or 4) I'd be doing so. $\endgroup$ – Scott Morrison♦ Mar 18 '10 at 18:08

5$\begingroup$ Closed. I know, I know, it's nice to have some questions like this to waste some time on, but we really don't need to have this at the top of the home page for several days. I'd recommend going to vote up this feature request meta.stackexchange.com/questions/3414/… if you happen to have meta.SE rep, and feel the same way. $\endgroup$ – Scott Morrison♦ Mar 19 '10 at 6:13
There is a famous anecdote about Barry Mazur coming up with the worst notation possible at a seminar talk in order to annoy Serge Lang. Mazur defined $\Xi$ to be a complex number and considered the quotient of the conjugate of $\Xi$ and $\Xi$: $$\frac{\overline{\Xi}}{\Xi}.$$ This looks even better on a blackboard since $\Xi$ is drawn as three horizonal lines.

16$\begingroup$ The full story involves a tshirt that Barry had commissioned and worn for the occasion... $\endgroup$ – Pete L. Clark Mar 18 '10 at 15:44

10$\begingroup$ @Pete: Can you fill us in with the full story? $\endgroup$ – Kevin H. Lin Mar 18 '10 at 19:20

26$\begingroup$ I should mention that I am a Mazur student, hence a designated keeper of the flame of Mazur lore (although there are about 50 other such people) but that I was not there to see the event myself. (Actually I never met Lang, to my regret.) The most authoritative description I have found is given by Paul Vojta here: ams.org/notices/200605/fealang.pdf $\endgroup$ – Pete L. Clark Mar 18 '10 at 21:06

12$\begingroup$ I had heard two additional versions of this story that differed from Vojta's in the ending, in the sense that Vojta's Serge didn't make a peep, and the other two Serges exploded. $\endgroup$ – S. Carnahan♦ Mar 19 '10 at 5:06

11$\begingroup$ @PeteL.Clark: About a year ago I was at a reception with Mazur and Rohrlich, and asked Mazur what had really happened during that lecture. What he described matched Vojta's description, except that Lang did not keep quiet. Instead, he said (calmly) to Mazur "You really seem to have a thing for horizontal lines". Since the intended reaction had not been produced, Mazur then opened the box containing the shirt and gave it to Serge. $\endgroup$ – KConrad Jul 14 '14 at 4:08
My favorite example of bad notation is using $\textrm{sin}^2(x)$ for $(\textrm{sin}(x))^2$ and $\textrm{sin}^{1}(x)$ for $\textrm{arcsin}(x)$, since this is basically the same notation used for two different things ($\textrm{sin}^2(x)$ should mean $\textrm{sin}(\textrm{sin}(x))$ if $\textrm{sin}^{1}(x)$ means $\textrm{arcsin}(x)$).
It might not be horrible, since it rarely leads to confusion, but it is inconsistent notation, which should be avoided in general.

59$\begingroup$ I take issue with "it rarely leads to confusion". Among mathematicians that's true, but among calculus students is another story. $\endgroup$ – Mark Meckes Mar 18 '10 at 18:54

18$\begingroup$ I refuse to use the notation $\sin^{1}(x)$, except perhaps to complain about it, and use the notation $\arcsin(x)$. This means that anything I do with nontrivial powers or inverses of trig functions use a different convention to the rest of any work I do with functions, but it is better than the ugliness of a notation with a single function which is inconsistent. $\endgroup$ – Niel de Beaudrap Mar 18 '10 at 22:24

36$\begingroup$ In my first semester of college I was marked wrong on an exam for writing $\sin^{1}(x)$ on a question whose answer was $\arcsin(x)$. When I complained, the professor kept explaining why $1/\sin(x)$ was incorrect. He claimed never to have heard of this notation for arcsin, and relented only after I got a second opinion from another professor in the department. So it's not only calculus students who get confused by this! $\endgroup$ – Tom Church Mar 19 '10 at 1:40

12$\begingroup$ Actually, I take issue with both of these notations. :) As a dynamicist, I would indeed agree that $\sin^2$ should be the second iterate of sine. However, $\sin$ is not invertible, and hence $\sin^{1}$ should not be used for the arcsine, which is only a specific branch of the inverse function. $(\sin_{[\pi/2,\pi/2]})^{1}$ would be ok I suppose ... $\endgroup$ – Lasse RempeGillen Jan 26 '15 at 16:29

5$\begingroup$ Why not writing $f^{\circ 2} = f \circ f$ and $f^{\circ 1}$? And $f^2$, $f^{1}$ are defined pointwise as usual. $\endgroup$ – Martin Brandenburg Sep 13 '15 at 8:25
I personally hate the notation $x \mid y$, for "$x$ divides $y$". Of course, I'm used to reading it by now, but a general principle I follow and recommend is:
Never use a symmetric symbol to denote an asymmetric relation!

46$\begingroup$ How do you cope with wedge products? Or tensor products? or, gasp, multiplication in a noncommutative group? I hope I didn't just blow your mind. :p $\endgroup$ – Willie Wong Mar 18 '10 at 16:55

15$\begingroup$ Graham, Knuth, and Patashnik, in "Concrete Mathematics", share your unease. They write x\y for "x divides y". $\endgroup$ – Michael Lugo Mar 18 '10 at 17:08

48$\begingroup$ Regarding wedge, tensor, multiplication symbols  I specifically said "relation", not "operation". And I stand by what I said  I cope just fine! :) $\endgroup$ – Marty Mar 18 '10 at 17:55

22$\begingroup$ Yes, when I tell undergraduates that x divides y means y/x is an integer I can see them thinking, "No way, he's messing with us." Then I tell them that actually 0 divides 0 as well, and I can practically see the steam coming out of their ears. (Come to think of it, this gives a funny variation on the old precalculus brainteaser "What is zero over zero"? Answer: "Yes.") $\endgroup$ – Pete L. Clark Mar 18 '10 at 21:15

32$\begingroup$ I also don't like the notation $x \backslash y$ for "x divides y". That seems likely to confuse students due to its resemblance to an operation. If I were to choose a symbol, I might use $x \triangleleft y$, to mean "x divides y". It preserves the vertical line, which we're all used to, and adds a directional component which I find appropriate. $\endgroup$ – Marty Mar 18 '10 at 22:18
I never liked the notation ${\mathbb Z}_p$ for the ring of residue classes modulo $p$. At one point, it confused the hell out of me, and this confusion is easily avoided by writing $C_p$, $C(p)$ or ${\mathbb Z}/p$.

13$\begingroup$ Yep. $\mathbb{Z}_p$ is the ring of $p$adic integers. But it took me some time to learn that too... $\endgroup$ – darij grinberg Mar 18 '10 at 16:06

23$\begingroup$ There is a slight ambiguity here. My personal convention is that C_p and C(p) refer to the cyclic groups with no mention of their ring structure, and it is Z/pZ (or even better, F_p) which refers to the actual ring. $\endgroup$ – Qiaochu Yuan Mar 18 '10 at 17:01

23$\begingroup$ Please don't confuse the residueclass ring ${\mathbb Z}_q$ (aka ${\mathbb Z}/q{\mathbb Z}$) with a Galois field of $q$ elements, conveniently denoted by ${\mathbb F}_q$ (or $GF(q)$). It is one of the most popular fallacies of our students to assume that both symbols denote the same mathematical object, even if $q$ is not a prime. $\endgroup$ – MRA Mar 18 '10 at 17:01

15$\begingroup$ You can never go wrong with $\mathbb{Z}/p\mathbb{Z}$ or $\mathbb{Z}/\mathfrak{p}$. $\endgroup$ – Harry Gindi Mar 18 '10 at 18:46

27$\begingroup$ Jim, I disagree. It makes sense if we agree that it should make sense. It's unambiguous. And it follows a noble tradition of ignoring the distinction between elements of a ring and the principal ideals that they generate. $\endgroup$ – James Cranch Jan 8 '12 at 0:56
Physicist will hate me for this, but I never liked Einstein's summation convention, nor the famous bra ($\langle\phi$) and ket ($\psi\rangle$) notation. Both notations make easy things look unnecessarily complicated, and especially the braket notation is no fun to use in LaTeX.

253$\begingroup$ Personally, I strongly dislike the misuse of the relation symbols
$<$
and$>$
instead of the appropriate$\langle$
and$\rangle$
angle brackets. I dislike it so strongly, in fact, that I edited this answer. $\endgroup$ – Harald HancheOlsen Mar 18 '10 at 19:16 
25$\begingroup$ I would vote up the previous comment seven times if I could. $\endgroup$ – Mark Meckes Mar 18 '10 at 19:50

13$\begingroup$ You can use latex's newcommand to make the typesetting easier. Type \newcommand{\bra}[1]{\langle #1} at the top of your document, and \bra\psi will produce $\newcommand{\bra}[1]{\langle #1} \bra\psi$ $\endgroup$ – userN Mar 18 '10 at 20:10

23$\begingroup$ Dirac notation isn't really justifiable for generic vectors on their own. But it is so helpful for quantum information! It is so much easier to read/write standard basis vectors as $a_1 a_2 \cdots a_n\rangle$ than as $\mathbf{e}_{a_1,a_2,\ldots,a_n}$ (or worse: $\mathbf{e}_{a_1} \otimes \cdots \otimes \mathbf{e}_{a_n}$). I would go further, and introduce this notation into introductions to probability. For expressing states of configuration spaces over distinguishable labels, it is quite good. $\endgroup$ – Niel de Beaudrap Mar 18 '10 at 22:34

15$\begingroup$ I would say the Einstein summation notation makes complicated things look misleadingly easy. When I write that scalar curvature is $R=g^{ij}R^k_{ikj}$, is it clear how much is actually going on there? It does prevent you from writing anything coordinatedependent, which is quite nice. $\endgroup$ – Tom Church Mar 19 '10 at 1:53
Mathematicians are really quite bad when it comes to notation. They should learn from programming langauges people. Bad notation actually makes it difficult for students to understand the concepts. Here are some really bad ones:
 Using $f(x)$ to denote both the value of $f$ at $x$ and the function $f$ itself. Because of this students in programming classes cannot tell the difference beween $f$ (the function) and $f(x)$ (function applied to an argument).
 When I was a student nobody ever managed to explain to me why $dy/dx$ made sense. What is $dy$ and what is $dx$? They're not numbers, yet we divide them (I am just giving a student's perspective).
 In Langrangian mechanics and calculus of variations people take the partial derivative of the Lagrangian $L$ with respect to $\dot q$, where $\dot q$ itself is the derivative of momentum $q$ with respect to time. That's crazy.
 The summation convention, e.g., that ${\Gamma^{ij}}_j$ actually means $\sum_j {\Gamma^{ij}}_j$ is useful but very hard to get used to.
 In category theory I wish people sometimes used any notation as opposed to nameless arrows which are introduced in accompanying text as "the evident arrow".

34$\begingroup$ Your first point is not a problem with the notation but with the users of the notation... $\endgroup$ – Mariano SuárezÁlvarez Mar 18 '10 at 15:23

88$\begingroup$ The Leibniz notation dy/dx for the derivative is perhaps the most clever bit of notation ever invented in the whole history of mathematics. $\endgroup$ – Franz Lemmermeyer Mar 18 '10 at 15:51

42$\begingroup$ Einstein summation is indeed retarded, not because it is hard to get used to, but because it makes it impossible to refer to a single addend rather than to the whole sum (as in: "each of the addends is nonnegative, hence the sum is nonnegative"), forcing writers to leave out parts of their arguments just because they can't write it with Einstein summation. Although someone has proposed a sum sign crossed out to mean "this is NOT a sum". $\endgroup$ – darij grinberg Mar 18 '10 at 15:58

39$\begingroup$ The Einstein summation convention is beautiful. It essentially makes it impossible to write down something that isn't coordinate independent. Or rather it's Penrose's abstract index notation that does this. But as his notation is essentially identical to Einstein's, there's not much difference. en.wikipedia.org/wiki/Abstract_index_notation (@andrej I thought you'd appreciate that kind of "type safety".) (@darij The fact that you can't refer to individual components is a virtue. It encourages people to state and write proofs that are independent of choice of basis.) $\endgroup$ – Dan Piponi Mar 18 '10 at 16:14

29$\begingroup$ In my first calculus class, a very big deal was made to make sure students understood that dy/dx was NOT a quotient; dy and dx had NO meaning whatsoever on their own, it was just notation. And then I got to differential equations, and on the first day the professor said "Now multiply by dx." The other students seemed perfectly OK with this, but it confused the heck out of me for a while. Maybe I was the only one who had actually believed the calculus professor that dx had no independent meaning. $\endgroup$ – Michael Benfield Mar 18 '10 at 19:45
I think composition of arrows $f:X\to Y$ and $g:Y\to Z$ should be written $fg$ not $gf$. First of all it would make the notation $\hom(X,Y)\to\hom(Y,Z)\to \hom(X,Z)$ much more natural: $\hom(E,X)$ should be a left $\hom(E,E)$ module because $E$ is on the left :) Secondly, diagrams are written from left to right (even stronger: Almost anything in the western world is written left to right). And i think the strange (1) needed when shifting complexes is an effect of this twisted notation.

10$\begingroup$ To some extent I agree. The problem is that the order in which composition is written in standard notation is the opposite of the temporal order, which by any standard is the "natural" one  first do this, then do this, then do this. In fact, I think it's reasonable to argue that the basis of the standard definition of function is the arrow of time. $\endgroup$ – Qiaochu Yuan Mar 18 '10 at 16:58

7$\begingroup$ Harald it was Herstein, the author of our favourite algebra book! $\endgroup$ – Maharana Mar 18 '10 at 17:41

5$\begingroup$ Jacobson's book on Lie algebras also uses $xf$ to mean $f(x)$. It seems that at the time this notation was all the rage. I find it very confusing now. $\endgroup$ – José FigueroaO'Farrill Mar 18 '10 at 18:34

18$\begingroup$ $xf$ makes a lot of sense when you realize that $x: 1 \rightarrow X$ and $f: X \rightarrow Y$ are both functions. $\endgroup$ – Steven Gubkin Mar 19 '10 at 13:55

6$\begingroup$ I believe the primary source of confusion here is that there is no reason whatsoever for the arrows to go from left to right. If by chance it would begin with something like $f:Y\leftarrow X$... $\endgroup$ – მამუკა ჯიბლაძე Jun 7 '15 at 5:17
My candidate would be the (internal) direct sum of subspaces $U \oplus V$ in linear algebra. As an operator it is equivalent to sum but with the side effect of implying that $U \cap V = \lbrace 0\rbrace$. Whenever I had a chance to teach linear algebra I found this terribly confusing for students.

7$\begingroup$ Shouldn't it imply
$U \cap V = \{0\}$
? I guess that's another piece of bad notation: not all trivial things equal $\varnothing$. $\endgroup$ – François G. Dorais♦ Mar 19 '10 at 0:10 
1$\begingroup$ @Francois: Sheesh, of course. Sorry. Fixed. $\endgroup$ – Alon Amit Mar 19 '10 at 0:20

1$\begingroup$ This isn't confusing as long as you clearly distinguish interior and exterior direct sums. $\endgroup$ – Qiaochu Yuan Mar 19 '10 at 5:16

6$\begingroup$ @Qiaochu: I'm really only talking about the interior case. It's then that the \osum is the same subspace as the sum but with the implied additional condition on the subspaces. $\endgroup$ – Alon Amit Mar 19 '10 at 7:16

1$\begingroup$ What would you do instead, write $U + V$ and note that $U \cap V = \{0\}$ on the side? I imagine that you might have to be more explicit (at least in introductory material) with arguments of the form ‹Because $U \cap V = \{0\}$, $U + V \cong U \oplus V$, so ….›. (I'm not claiming that this is a bad thing!) $\endgroup$ – Toby Bartels Feb 27 '17 at 14:43
The notation ]a,b[ for open intervals and its ilk. Sorry, Bourbaki.

39$\begingroup$ I like ]a,b[ because it does not have the ambiguity of (a,b) and it's very clear that a,b are excluded. $\endgroup$ – Martin Brandenburg Apr 16 '10 at 15:07

26$\begingroup$ Personally, i have nothing against $]a,b[$; it just makes the `parenthesis (bracket) matcher' of my text editor blush. $\endgroup$ – Suvrit Oct 17 '10 at 8:47

28$\begingroup$ Imagining $a$ and $b$ on a real line, the notation $]a,b[$ looks to me more like $\mathbb R \setminus (a,b)$... $\endgroup$ – Y. Pei Feb 5 '15 at 18:48

17$\begingroup$ Find all $(a,b)$ such that $(a,b)\in(a,b)$ $\endgroup$ – Thomas Rot Oct 14 '15 at 0:23

16$\begingroup$ @LSpice: The first $(a,b)$ is an ordered pair, the second $(a,b)$ is the gcd, the third $(a,b)$ is an interval. I leave it to you to solve the exercise. $\endgroup$ – Thomas Rot Nov 3 '15 at 17:07
Writing a finite field of size $q$ as $\mathrm{GF}(q)$ instead of as $\mathbf{F}_q$ always rubbed me the wrong way. I know where it comes from (Galois Field), and I think it is still widely used in computer science, and maybe in some allied areas of discrete math, but I still dislike it.

9$\begingroup$ Well, maybe $\mathrm{GF}(q^{q^2})$ is still better than $\mathbb F_{q^{q^2}}$. Indices are nice until one has to nest three or more of them... $\endgroup$ – darij grinberg Mar 18 '10 at 20:10

5$\begingroup$ In references authors may write exp($z$) instead of $e^z$ when the exponent could get particularly complicated, but when does this ever really happen for sizes of finite fields? If I find it necessary to write something like ${\mathbf F}_{q^{q^2}}$ again, I'll reconsider my opposition to writing GF. :) $\endgroup$ – KConrad Mar 18 '10 at 21:20

4$\begingroup$ Best thing is when it's (re)translated as "GaloisFeld" in German. $\endgroup$ – user717 Mar 18 '10 at 23:36

3$\begingroup$ The worst thing about "GaloisFeld" is that it has apparently been the standard notion in German in the beginning of the 20th Century (at least, Witt uses it).  @KConrad: "Contributions to the Theory of Finite Fields" by Oystein Ore (Transactions of the American Mathematical Society, Vol. 36, No. 2. (Apr., 1934), pp. 243274) works with $\mathbb{F}_{p^{ff^{\prime}}}$, which he fortunately abbreviates. Otherwise, I think not even JSTOR's high scanning quality would suffice to recognize the symbols. $\endgroup$ – darij grinberg Mar 18 '10 at 23:54

1$\begingroup$ @darij: Are you sure? "Field" is "Körper" in German and always has been. I don't know why it's called "field" in English, but it's okay. Only when people then translate "Galois field" as "GaloisFeld", I think it's retarded... $\endgroup$ – user717 Mar 19 '10 at 12:03
As Trevor Wooley used to always say in class, ``Vinogradov's notation sucks....the constants away."
For those who don't know, Vinogradov's notation in this context is $f(x)\ll g(x)$ meaning $f(x) = O(g(x)).$ (if you prefer bigO notation, that is).

18

12$\begingroup$ For the Big O notation "sucks" and Vinogradov's notation has sense and. Say $o(x)=O(x)$ but $O(x)\not=o(x)$. $\endgroup$ – Anton Petrunin Mar 18 '10 at 23:40

15$\begingroup$ The above is not a problem caused by bigO notation, but a problem caused by failing to recognize that $O(x)$ is a set. The correct equivalent statements are that $o(x) \subseteq O(x)$ but $O(x) \not\subseteq o(x)$. [Of course, $o(x) \ne O(x)$ is also correct.] As equality is the most important relation we have, one should never write an equals sign unless one really means it! $\endgroup$ – Niel de Beaudrap Mar 19 '10 at 6:24

26$\begingroup$ So $f(x) < \!\! < f(x)$? That's terrible. Weak inequalities should have a horizontal line in their notations, like $\leq$ and $\subseteq$. $\endgroup$ – David E Speyer Mar 19 '10 at 7:12

7$\begingroup$ @LSpice I've actually taken to writing $\subseteq$ or $\subsetneq$ whenever the issue is important and not obvious from context. While I agree that $\subset$ should mean $\subsetneq$, there is not enough consensus on this point to be sure I am understood. $\endgroup$ – David E Speyer Oct 13 '15 at 20:40
I rather dislike the notation $$\int_{\Omega}f(x)\,\mu(dx)$$ myself. I realize that just as the integral sign is a generalized summation sign, the $dx$ in $\mu(dx)$ would stand for some small measurable set of which you take the measure, but it still rubs me the wrong way. Is it only because I was brought up with the $\int\cdots\,d\mu(x)$ notation? The latter nicely generalizes the notation for the Stieltjes integral at least.

6$\begingroup$ I also used to dislike that notation, but I think it's only a matter of upbringing. It's useful, e.g., for dealing with things like Markov kernels: when you have $K:\Omega\times\mathcal{F}\to\mathbb{R}$, where $\mathcal{F}$ is a $\sigma$algebra on $\Omega$ and for each $x$, $K(x,\cdot)$ is a measure, you can easily write things like $\int f(x)K(y,dx)$, which is much harder to write with the $\int f(x) d\mu(x)$ notation. $\endgroup$ – Mark Meckes Mar 18 '10 at 19:59

1$\begingroup$ Also, whether $\int f(x) d\mu(x)$ nicely generalizes the Stieltjes integral notation or not depends on your viewpoint. If $\frac{d\mu}{dx} = h = \frac{d H}{dx}$, then indeed $\int f(x) d\mu(x) = \int f(x) dH(x)$. On the other hand, in that case $\mu=h$ (not $H$) as distributions, i.e. $\langle f,\mu \rangle = \langle f,h \rangle$. $\endgroup$ – Mark Meckes Mar 18 '10 at 20:05

3$\begingroup$ That is one of the worse. Formally you have to write $(f\mu)(\Omega)$, but no one does it. It should be form under the integral, so in principle you can think that measure is a form so you can write $\int f\mu$ and no one does it either... Instead everyone writes something which has no sense like $\int f\mu(dx)$ or $\int fd\mu$... – Anton Petrunin 0 secs ago $\endgroup$ – Anton Petrunin Mar 20 '10 at 16:44

2$\begingroup$ The $d$ is entirely superfluous in $\int \cdots d\mu(x)$ (which does not generalize the Stieltjes integral, because the measure in $\int \cdots dg(x)$ is $dg$, not $g$). You should either write $\int \cdots \mu(dx)$ or (if you're willing to defy convention) $\int \cdots \mu(x)$ (or $\int \cdots \mu$ if there is no need to use a dummy variable). You can see this in action at ncatlab.org/nlab/show/measure+space. $\endgroup$ – Toby Bartels Feb 27 '17 at 15:07
I get very frustrated when an author or speaker writes "Let $X\colon= A\sqcup B$..." to mean:
 $A$ and $B$ are disjoint sets (in whatever the appropriate universe is),
 and let $X\colon= A\cup B$.
If they just meant "form the disjoint union of $A$ and $B$" this would be fine. But I've seen speakers later use the fact that $A$ and $B$ are disjoint, which was never stated anywhere except as above. You should never hide an assumption implicitly in your notation.

3$\begingroup$ But the point is that there is no universally agreed upon definition of what the "disjoint union" of two nondisjoint sets is. Or is there? (I suppose if you forced me to give one, I would put
$A \coprod B = A \times \{0\} \cup B \times \{1\}$
, but surely not everyone agrees with this?) $\endgroup$ – Pete L. Clark Mar 19 '10 at 6:18 
11$\begingroup$ This is very similar to Alon Amit's post about writing internal direct sums using the more general direct sum notation without mention of the implicit assumptions. The problem is when the notation does not refer to the "disjoint union" of notnecessarilydisjoint sets, but rather to a union of two sets that are assumed implicitly to be disjoint. $\endgroup$ – Jonas Meyer Mar 19 '10 at 9:28

32$\begingroup$ Oh for Pete's sake! :D Seriously, Pete: what's the problem? The disjoint union is defined up to unique isomorphism by a universal property; the particular settheoretic encoding is of no importance whatever. $\endgroup$ – Todd Trimble♦ Nov 19 '12 at 2:41

3$\begingroup$ I agree with Tom Church that the cited usage is inept. $\endgroup$ – Todd Trimble♦ Nov 19 '12 at 2:44

3$\begingroup$ I agree with Todd Trimble. Also, notice that there is no notion of "two sets are disjoint" for a category theorist. It only makes sense to say that two arrows $A \to X \leftarrow B$ are disjoint. $\endgroup$ – Martin Brandenburg Sep 13 '15 at 8:33
The use of squared brackets $\left[...\right]$ for anything. It's not bad per se, but unfortunately it is used both as a substitute for $\left(...\right)$ and as a notation for the floor function. And there are cases when it takes a while to figure out which of these is meant  I'm not making this up.
The word "character" meaning: a 1dimensional representation, a representation, a trace form of a representation, a formal linear combination of representations, a formal linear combination of trace forms of representations.
The word "adjoint", and the corresponding notation $A\mapsto A^{\ast}$, having two completely unrelated meanings.

3$\begingroup$ Do people really use squared brackets for the floor function? I thought it has its own symbol (one that makes a lot of sense at that). Also, if you want to talk about words that are abused, at least be outraged by "normal"! $\endgroup$ – Willie Wong Mar 18 '10 at 16:16

21$\begingroup$ Indeed, the floor function should be written
$\lfloor\cdot\rfloor$
. Pet peeve of mine. $\endgroup$ – Harald HancheOlsen Mar 18 '10 at 16:24 
2$\begingroup$ Yes, squared brackets are still used in some parts of the world for floor, unfortunately. And as for "normal", it indeed belongs into the list, though it's not as bad as people claim; the different uses of "normal" mostly belong to different fields of mathematics, and thus it's not that easy to confuse them. Except "normal" for Hopf algebras vs. "normal" for commutative rings. $\endgroup$ – darij grinberg Mar 18 '10 at 16:33

4$\begingroup$ @darij: Take a topological group $G$, now consider a normal subgroup $H$ ... $\endgroup$ – Gerald Edgar Mar 18 '10 at 18:54

1$\begingroup$ Wasn't there a topology textbook that had two incompatible definitions of "perfectly normal"? (I think it was a translation and the original used different words for "perfect", but still...) $\endgroup$ – François G. Dorais♦ Mar 19 '10 at 0:31
The term "symplectic group" used to mean the group $U(n,{\mathbb H})$. It's as if people called $U(n)$ and $GL(n,{\mathbb R})$ by some single name.

2$\begingroup$ Allen, that's amazing, and I'd never heard of it! Where did it happen? $\endgroup$ – LSpice Sep 8 '10 at 4:05
My personal pet peeve of notation HAS to be algebraists writing functions on the right a la Herstein's "Topics In Algebra". I don't know why they do it when everyone else doesn't. I think one of them got up one day and decided they wanted to be cooler then everyone else, seriously...

31$\begingroup$ The rule isn't strange, but it is an example of “rationalization” efforts going too far. Like having 100 degrees (or “grads”) in a right angle, or my favourite, the 13 month calendar. The idea is quite clever, actually: 13 months, each 28 days long, add up to 364 days. The 365th day (and the 366th, in leap years) should be a universal holiday, and it should not be assigned a weekday name! Thus the calendar for every month would look the same – from Monday the 1st to Sunday the 28th. But thinking you can actually inflict this on society verges on lunacy. $\endgroup$ – Harald HancheOlsen Mar 18 '10 at 19:29

46$\begingroup$ Come on, don't tell me that the "lunacy" pun in the context of the 13 months calendar was unintentional... $\endgroup$ – darij grinberg Mar 18 '10 at 20:11

14$\begingroup$ The "functions on the right" style is alive and well... in many object oriented programming languages. Methods are functions applied to objects. $\endgroup$ – Niel de Beaudrap Mar 18 '10 at 22:07

5$\begingroup$ Some genius among the programmers of GAP must have deemed it a great idea to drag that unholy right function notation out of its welldeserved grave and force it upon users. To make things worse, $f(x)$ is written x^f in GAP, which means additional fun because of the way systems process the tilde sign. And, of course, it is a consequence that group multiplication on $S_n$ in GAP is opposite to the rest of the world. Dear Mr. Cool, thanks for proving once again that open source software is not written for users. $\endgroup$ – darij grinberg Sep 12 '10 at 23:34

11$\begingroup$ I once had the pleasure of teaching a course ("applied modern algebra") from a textbook that used the "composition on the right" convention for binary relations, defined functions as a special case of binary relations but used composition on the left for them, and defined permutations as a special case of functions but used composition on the right for them. $\endgroup$ – Andreas Blass Mar 30 '11 at 16:22
I don't like (but maybe for a bad reason) the notation $F\vdash G$ for $F$ is left adjoint to $G$.
Any comment ?

2$\begingroup$ If we ignore the extra information (unit and counit), then adjointness is a binary relation. Presumably then you want to use the notation which places a symbol between F and G. What is wrong with $\dashv$? Would you prefer another symbol? $\endgroup$ – Andrej Bauer Mar 18 '10 at 15:24

22$\begingroup$ Isn't it $F\dashv G$ for "F left adj. to G"? $\endgroup$ – Gerrit Begher Mar 18 '10 at 15:33

9$\begingroup$ I think Garief's question suggests why this is bad notation: I doubt I could ever remember which way it is supposed to go without checking wikipedia. Either convention would make sense since F is on the left. Using a symmetric symbol here is definitely a bad idea, but when I need to abbreviate "F is left adjoint to G" I often write something like "F:D <> C:G is an adjunction". $\endgroup$ – Sam Lichtenstein Mar 18 '10 at 17:09

5$\begingroup$ The only way I can remember which one is which is by looking at $Hom(F(\cdot),\cdot) \cong Hom( \cdot ,G(\cdot))$ $\endgroup$ – Harry Gindi Mar 18 '10 at 18:49

6$\begingroup$ I recently realized that the mnemonic for the unit and counit is that the left (resp. right) adjoint occurs visually first when it occurs on the left (resp. right) side of the arrow. So if F is left adjoint to G then the unit and counit are 1 > GF and FG > 1. $\endgroup$ – Sam Lichtenstein Mar 18 '10 at 23:53
A cute idea but for which I have yet to find supporters is D. G. Northcott's notation (used at least in [Northcott, D. G. A first course of homological algebra. Cambridge University Press, London, 1973. xi+206 pp. MR0323867) for maps in a commutative diagram, which consists in enumerating the names of the objects placed vertices along the way of the composition. Thus, if there is only one map in sight from $M$ to $N$, he writes it simply $MN$, so he has formulas looking like $$A'A(ABB'') = A'ABB'' = A'B'BB'' = 0.$$ He also writes maps on the right, so his $$xMN=0$$ means that the image of $x$ under the map from $M$ to $N$ is zero.
I would not say this is among the worst notations ever, though.
Students have big difficulties when first confronted with the $o(\cdot)$ and $O(\cdot)$ notation. The term $o(x^3)$, e.g., does not denote a certain function evaluated at $x^3$, but a function of $x$, defined by the context, that converges to zero when divided by $x^3$.

6$\begingroup$ And for the littleoh notation in particular, one often forgets to specify the limit. There is a rather substantial difference between “
$o(x^3)$
as$x\to\infty$
” and “$o(x^3)$
as$x\to0$
”, after all. $\endgroup$ – Harald HancheOlsen Mar 18 '10 at 19:36 
4$\begingroup$ @exfalsoquodlibet: How would you prefer to express these concepts? It is possible to express these concepts quite nicely as limits of ratios, but the bigO, littleo notation has the advantage of being fairly clear while expending less effort  which is ultimately the goal of notational devices. $\endgroup$ – Niel de Beaudrap Mar 18 '10 at 22:16

9$\begingroup$ exfalsoquodlibet Big O notation is useful because I can include it as a term in larger expressions. For example,
$$\sum \log(1/(1\chi(p)/p^s)) = \sum \left( \chi(p)/p^s + O(1/p^{2s}) \right) = \sum \chi(p)/p^s + O(1)$$
as $s \to 1^{+}$. Try writing that in your preferred notation and see which is more readable. $\endgroup$ – David E Speyer Mar 19 '10 at 7:19 
2$\begingroup$ I find the only thing wrong with the big O notation is the equality symbol: $f = O(g)$. It's really a reflexive transitive relation, so one should write $f \leq_O g$ or something. $\endgroup$ – Todd Trimble♦ Feb 26 '13 at 1:35

4$\begingroup$ @StevenStadnicki Very belated response, but: yes, that would be a very proper use of notation. In fact there's a notation pun that I like: if we think of Hardy fields that consist of germs of functions at infinity as valuation fields $K$, and remember that in algebraic geometry one often uses $O$ or $\mathcal{O}$ for the local valuation ring of germs that are bounded at the point (in this case $\infty$), then your $f \in O(g)$ literally means $f \in O g$ in the sense of principal fractional ideals (the same as $f \leq g$ in the ordered valuation group $K^\ast/O^\ast$). $\endgroup$ – Todd Trimble♦ Oct 13 '15 at 22:15
I have struggled with 'dx'. I've spent years trying to study every different approach to calculus that I could find to try and make sense of it. I read about the limit definitions in my first book, vector calculus with them as pullbacks of linear transformations or flows/flux, differential forms from the bridge project, kforms, nonstandard analysis which enlarges $\mathbb{R}$ to give you infinitesimals (and unbounded numbers) but the same first order properties and lets integral be defined as a sum, constructive analysis using a monad to take the closure of the rationals to give reals... but I am still just as confused as ever, I understand that the mathematical notation doesn't have a compositional semantics but still don't really get it  one of the problems is despite not really understanding it, or having any abstract definition of it.. I can still get correct answers and I really hope this doesn't become a theme as I study more topics in mathematics.

4$\begingroup$ Almost always, dx refers to the 1form obtained by exterior differentiation from the function x on your space (say the graph of a function R>R viewed as a subset of R^2, in which case x is the projection onto the first factor). It's not really mysterious. I don't know about nonstandard analysis, but I can imagine it would mean something else in such contexts. Although in my opinion, authors would do well to use a new symbol to denote something like "infinitessimal change in x" (assuming they can make mathematical sense of this) rather than overloading a symbol with a perfectly good meaning $\endgroup$ – Sam Lichtenstein Mar 18 '10 at 17:18

1$\begingroup$ I'm voted 1 for this remark? Is it because it's too basic or boring a problem or why? :/ $\endgroup$ – muad Mar 18 '10 at 18:22

4$\begingroup$ Probably you were downvoted because you did not really explicitely explain the confusion caused by dx. As somebody who uses differential forms a lot I have never stumbled over a situation where dx was mysterious for me. $\endgroup$ – J Fabian Meier Mar 18 '10 at 18:56

6$\begingroup$ There are definitely some annoying things about the $dx$ or $\frac d{dx}$ notation, but I think Newton's notation was worse, so $dx$ can't be the worst. $\endgroup$ – Douglas Zare Mar 18 '10 at 18:56

18$\begingroup$ Then how about $dS=\sqrt{dx^2+dy^2}$ (<some physicists love this) ?!... :) $\endgroup$ – M.G. Mar 18 '10 at 20:26
p < q as in "the forcing condition p is stronger than q".

$\begingroup$ Think of intervals: A smaller interval contains more information in the sense that it determines an arbitrary element with less error. Hence giving the smaller interval is the "stronger" condition. (If I remember correctly, there really is a forcing notion where this is literally true. The poset it choosen to be the Borel sets of [0,1] or something similar. Please correct me if that's wrong, I've never really had anything to do with forcing) $\endgroup$ – Johannes Hahn Dec 1 '13 at 1:02

$\begingroup$ I don't think it is an issue, since this is the appropriate ordering in the Boolean completion, where the weakest condition, 1, is the largest. Anyway, Matt Foreman avoids having to decide whether to use this or the other convention by writing $p\Vdash q $ and only using separative posets. $\endgroup$ – Andrés E. Caicedo Jul 12 '17 at 23:15
I hate the short cut $ab$ for $a\cdot b$. Everyone get used to it, BUT it creates very deep problem with all other notation; say you never can be sure what $f(x+y)$ or $2\!\tfrac23$ might be...
Also in modern mathematics people do not multiply things too often, so it does not have sense to make such a short cut.
Yet the shortcut $x^n$ is really bad one. One can not use upper indexes after this. It would be easy to write $x^{\cdot n}$ instead.

8$\begingroup$ Writing $a\dot b$ every time would make group presentation theorems a night mare to read. $\endgroup$ – Anonymous Mar 18 '10 at 21:01

5

13$\begingroup$ At the risk of stating the obvious, I will venture that this convention exists for a simple reason. In a tremendous number of situations, it makes parsing a mathematical expression roughly equivalent in effort to parsing a written sentence in a European language; thus reducing an important task to one which had been previously solved. $\endgroup$ – Niel de Beaudrap Mar 18 '10 at 22:40

8$\begingroup$ Can you elaborate what deep problems are created by this shortcut? $\endgroup$ – Andrea Ferretti Mar 19 '10 at 0:01

5$\begingroup$ BTW, I am impressed by number of negative votes  I think it only shows that we are not ready to admit that the notation we use everyday is the worst one. $\endgroup$ – Anton Petrunin Dec 1 '13 at 20:14