MathOverflow will be down for maintenance for approximately 3 hours, starting Monday evening (06/24/2013) at approximately 9:00 PM Eastern time (UTC-4).

## Suggestions for good notation

I occasionally come across a new piece of notation so good that it makes life easier by giving a better way to look at something. Some examples:

• Iverson introduced the notation [X] to mean 1 if X is true and 0 otherwise; so for example Σ1≤n<x [n prime] is the number of primes less than x, and the unmemorable and confusing Kronecker delta function δn becomes [n=0]. (A similar convention is used in the C programming language.)

• The function taking x to x sin(x) can be denoted by x ↦ x sin(x). This has the same meaning as the lambda calculus notation λx.x sin(x) but seems easier to understand and use, and is less confusing than the usual convention of just writing x sin(x), which is ambiguous: it could also stand for a number.

• I find calculations with Homs and ⊗ easier to follow if I write Hom(A,B) as A→B. Similarly writing AB for the set of functions from B to A is really confusing, and I find it much easier to write this set as B→A.

• Conway's notation for orbifolds almost trivializes the classification of wallpaper groups.

Has anyone come across any more similar examples of good notation that should be better known? (Excluding standard well known examples such as commutative diagrams, Hindu-Arabic numerals, etc.)

-
this question is broadly useful, so perhaps better as community wiki? – suVRit Oct 20 2010 at 20:09
In set theory we write ${}^B A$ for the set of functions from $B$ to $A$. – Andres Caicedo Oct 20 2010 at 20:54
I've always assumed that the notation $A^B$ is because of the "exponential law" $(A^B)^C = A^{B\times C}$ ... – Kevin Lin Oct 20 2010 at 23:18
Arabic numerals ? Ah yes, they were transmitted to Europe by the Arabs. – Chandan Singh Dalawat Oct 21 2010 at 3:29
Isn't $x \mapsto f(x)$ commonplace? As for homomorphisms, they are not simply maps, and $\mathrm{Hom}(A, B)$ denotes the whole class, while $A \to B$ denotes a single mapping. – Kallikanzarid Oct 21 2010 at 3:38

$\sin^{-1}(x)$ as opposed to $\text{arcsin}(x)$. This encapsulates the fact that it is an inverse function.

-
And, combined with $sin^2(x)$ for $(sin(x))^2$, it make one ponder the meaning of $sin^n(x)$ when $n=-1$. – Harald Hanche-Olsen Oct 20 2010 at 20:10
I have to go with this notation being anti-useful, since it is in fact not an inverse function except on a restricted domain (and that domain is different than the one for $\cos^{-1}$). Not that I have much call for it, but I decided once that I would never use $\sin^{-1}$. It takes greater fortitude to abandon $\sin^2$. – Ryan Reich Oct 20 2010 at 20:17
I have also permanently abandoned $\sin^{-1}$ after having numerous students mysteriously convert an $\arcsin$ into a $\csc$ without realizing it. (Of course, the notations $\sec$, $\csc$ and $\cot$ are almost completely worthless themselves.) – JBL Oct 20 2010 at 21:34
On this subject, $\arcsin(x)$ is actually great notation because it reminds you what the restricted domain is. That is, it gives outputs which are lengths of arcs (measured, as always, from the positive x-axis). Of course, this is never mentioned. – Ryan Reich Oct 20 2010 at 21:56
Gauss raged against the deplorable sin$^2x$ notation more than 150 years ago (sorry, I don't have the reference.) I guess we're stuck with it, though, since it's concise and ubiquitous in school maths. Also, how often do we really need the multiple composition of trig functions? – John Bentin Oct 21 2010 at 11:37

The notation $\perp$ to denote either orthogonality, or to indicate independent random variables, or perhaps even to indicate relatively prime numbers.

-
Wait, if $n \perp m$ means that n and m are relatively prime, then does $(n,m)$ represent their inner product? Far out... – Ryan Reich Oct 20 2010 at 20:19
With Knuth's proposal to write $k \perp n$ for k relatively prime to n' we can write by the Moebius inversion formula $\sum_{1 \le k \le n} \left[ k \perp n \right] = \sum_{1 \le k \le n} \left[ k \mid n \right] \mu\left(\frac{n}{k}\right).$ Note that by the conventions of the Iverson bracket when the Iverson-bracketed statement is false, it annihilates anything it is multiplied by - even if that other factor is undefined'. – Bruce Arnold Oct 22 2010 at 15:46
@David: I think it's appropriate for the "inner product" to be a (bi-)multiplicative function rather than an additive one, since "orthogonality" is a multiplicative concept. Note: "far out" was a term of appreciation! – Ryan Reich Oct 22 2010 at 22:03

String diagram-notation makes for example adjoint functors, monads, tensor categories,... much clearer.

-
And it also connects monoidal categories to braids, knots, links, tangles... – Qiaochu Yuan Oct 20 2010 at 20:45

$G \circlearrowleft X$ (or $G \circlearrowright X$) to denote that $G$ acts on $X$.

Edit by A.H. :
Here are some latex definitions that produce the symbol that David Speyer describes in his comment:

\def\acts{
\hspace{.1cm}
{
\setlength{\unitlength}{.30mm}
\linethickness{.09mm}
\begin{picture}(8,8)(0,0)
\qbezier(7,6)(4.5,8.3)(2,7)
\qbezier(2,7)(-1.5,4)(2,1)
\qbezier(2,1)(4.5,-.3)(7,2)
\qbezier(7,6)(6.1,7.5)(6.8,9)
\qbezier(7,6)(5,6.1)(4.2,4.4)
\end{picture}
\hspace{.1cm}
}}


and

\def\acted{
\hspace{.1cm}
{
\setlength{\unitlength}{.30mm}
\linethickness{.09mm}
\begin{picture}(8,8)(0,0)
\qbezier(1,6)(3.5,8.3)(6,7)
\qbezier(6,7)(9.5,4)(6,1)
\qbezier(6,1)(3.5,-.3)(1,2)
\qbezier(1,6)(1.9,7.5)(1.2,9)
\qbezier(1,6)(3,6.1)(3.8,4.4)
\end{picture}
\hspace{.1cm}
}}

-
I prefer $G \curvearrowright X$. – Mark Schwarzmann Oct 20 2010 at 20:34
I use a notation like that, but I have not found it in latex: the closest I can find is $G\subset X$ but with the $\subset$ turned into an arrow (so that $G$ appears as the label on the arrow, you could say) I distinctly dislike using \circlearrowleft for this! :) – Mariano Suárez-Alvarez Oct 20 2010 at 20:35
In hand writing, I prefer a roughly 3/4 circle which starts at about 4 oclock and ends at about 2 oclock. That way the arrow starts at X and returns there. Important note: if draw the arrow going the other way, people may mistake it for a capital G. – David Speyer Oct 20 2010 at 22:36
I agree with David Speyer. You want the arrow to begin and end at $X$ and go by way of $G$. (Not that this answers the original question, which was to come up with notation that substantially improves understanding...) – James Borger Oct 21 2010 at 1:39
\lcirclearrowright in MnSymbol looks ok. – Hsien-Chih Chang Oct 22 2010 at 1:22

Among recent introductions, I like the notation and names (introduced by Kenneth Iverson and popularized by Donald Knuth) for the ceiling function $\lceil x\rceil$ and floor function $\lfloor x\rfloor$. Compare with the heavy "approximation by excess/defect"...

-
According to Knuth, this notation and these names were introduced by Iverson in his book "A programming language" in 1962. – Richard Borcherds Oct 20 2010 at 20:25

The three-dot notation $f\mathrel{\scriptsize\vdots}A\to B$ to indicate that $f$ is a partial function from $A$ to $B$, meaning that $\text{dom}(f)\subseteq A$ rather than $\text{dom}(f)=A$. Partial functions are pervasive in logic, especially computability theory and set theory, and this notation is both compact and suggestive.

-
I use $\rightharpoonup$ for partial functions. – Andreas Blass Oct 20 2010 at 22:23
Use a broken arrow instead: f : A - - - > B. I have no idea how three dots is suggestive of the domain being something smaller than A. – KConrad Oct 21 2010 at 18:15
I just meant that it suggests that $f$ is something like a function from A to B, without being intrusive. This notation is really useful in situations where you have numerous partial functions of different arities running around. – Joel David Hamkins Oct 21 2010 at 18:22
Never saw the three-dot notation. It looks like a smudge or tiny dead gnat to me. I use the notation mentioned by Andreas. – Todd Trimble Oct 30 2010 at 18:28

$a \vee b$ and $a \wedge b$ to denote the maximum and minimum of the numbers $a$ and $b$. (This seems to be well-known only among probabilists.)

-
This notation is also used in a lattice or Boolean algebra for least-upper-bound and greatest-lower-bound, which agrees with your meaning in a total order, such as the reals. – Joel David Hamkins Oct 20 2010 at 21:01
Sometimes I have seen $a \cup b$ and $a \cap b$ for the greatest common divisor and least common multiple of positive integers $a, b$. Much better than $(a, b)$. – ABayer Oct 21 2010 at 3:59
AByer, it's by analogy with $\cup$ and $\cap$ that I remember that $\vee$ is maximum and $\wedge$ is minimum. (Otherwise I tend to think that since $\vee$ points down, it must be the minimum.) – Michael Lugo Oct 21 2010 at 4:20
Actually, I find this notation incredibly frustrating for the reason Michael mentions in his comment; $a\vee b$ is "clearly" a cartoon of a point lying below both $a$ and $b$. – JBL Oct 22 2010 at 13:19

I once came across the notation $\underline{n}$ for the set $\lbrace 1,2,\dots,n\rbrace$. It came in very handy to write $i \in \underline{n}$ instead of $1\leq i \leq n$ or $i \in \lbrace 1,2,\dots,n\rbrace$.

-
Sometimes people write ${}[n]$ for this. I agree, a shorthand of this kind is quite useful. – Andres Caicedo Oct 20 2010 at 20:51
In set theory, every ordinal number is equal to the set of smaller ordinal numbers (a definition due to von Neumann), and this makes for infinitely many conveniences similar to the ones you mention. Thus, n={0,1,...,n-1} and in general $\alpha=\{\beta | \beta\lt\alpha\}$. – Joel David Hamkins Oct 20 2010 at 21:04
This can be confusing though, since the topologist's simplex $[n]$ is the ordered set $\{0<...<n\}$. The rationale is that $n$ counts the number of arrows. – Harry Gindi Oct 21 2010 at 4:52
Also, if I'm not mistaken, enumerative algebraic geometers use $[n]$ to denote {$1, \cdots , n$}. – Qfwfq Oct 21 2010 at 11:35
@Andres Caicedo, see also Richard Stanley's answer: $\binom{X}{k}$ is a nice notation for the $k$-subsets of a set $X$, mirroring the binomial coefficient $\binom{x}{k}$. – JBL Oct 22 2010 at 13:22

I like $f\colon\thinspace M\looparrowright N$ to denote an immersion of smooth manifolds.

-
I use exactly this arrow to denote an operation of a group M on a space N. – Jan Weidner Oct 20 2010 at 21:39
I prefer the notation $\alpha: G \curvearrowright X$ for an action $\alpha$ of $G$ on $X$ (see my answer). – Qfwfq Oct 20 2010 at 22:57
... And I also like the "self intersecting arrow" to denote immersions that are possibly not embeddings. – Qfwfq Oct 20 2010 at 22:58

Instead of $[X]$ one often sees $\mathbf 1_{X}$ (especially in probability work?). This is neat because it literally is 1 on $X$. Also it has the advantage over $[X]$ that you can write things like $(2+\mathbf 1_{X})^2$ for the function that is $9$ when $X$ occurs and $4$ otherwise; $(2+[X])^2$ would be less appealing here. On the other hand, if there is a lot of notation replacing "$X$" this is not so good: $$\mathbf 1_{n_k\in \{n: n\text{ prime}\}}.$$

-
so how about $1[X]$? – Hsien-Chih Chang Oct 21 2010 at 2:25
$\mathbf{1}_X$ is conventionally used for characteristic function of a set. – Kallikanzarid Oct 21 2010 at 3:42
@Kallikanzarid: Good point, really it should be $\mathbf 1_X(\omega)=[\omega\in X]$. But if $\omega$ is an outcome in a probability experiment then often $\omega$ is suppressed, so instead of $\mathbf 1_X(\omega)$ one would write $\mathbf 1_X$. – Bjørn Kjos-Hanssen Oct 21 2010 at 3:58
The problem with 1_{X}, as you observed, is that it pushes the main stuff to the small font of the subscript. I don't understand the advantage you wrote of: what's wrong with (2+[X])²? IIRC, expressions of this sort are used without hesitation in Graham-Knuth-Patashnik (Concrete Mathematics). – shreevatsa Oct 28 2010 at 6:51

$D_j f$ to denote the partial derivative of a function between Euclidean spaces, w.r.t. the $j$'th coordinate. For some reason Jacobi's notation $\frac{\partial f}{\partial x_j}$ has become more popular. Jacobi's notation tends to cause much ambiguity and confusion, a point which is emphasized in the book "Multidimensional Real Analysis" by Duistermaat & Kolk. For instance (this example is taken from their book), let $e_1,e_2$ be the standard basis for $\mathbb{R}^2$ and define a new basis by $e'_1 = e_1 + e_2, e'_2 = e_2$. The passage from one basis to another is as follows: If $x_1 e_1 + x_2 e_2 = y_1 e'_1 + y_2 e'_2$ then $y_1 = x_1, y_2 = x_2 -x_1$. Now the meaning of $\frac{\partial y_2}{\partial y_1}$ is ambiguous: If one interprets $y_1$ and $y_2$ as independent coordinate functions, then $\frac{\partial y_2}{\partial y_1} = 0$. On the other hand, $\frac{\partial y_2}{\partial y_1} = \frac{\partial (x_2 -x_1)}{\partial x_1} = -1$, right? This was the source of much confusion for me when I was taught multivariate calculus and the notation $D_j f$ would have eliminated this confusion entirely.

-

To denote an action $\alpha: G\times X \rightarrow X$ of a group $G$ on a space $X$, there is the nice piece of notation:

$\alpha: G \curvearrowright X$

or simply

$G \curvearrowright X$ (the latter when the action is understood from the context).

E.g. you can say something like: $\rm{GL}(V) \curvearrowright V$ linearly. Or, to say that $W$ is an invariant subspace for $G \curvearrowright V$, you just write: $G \curvearrowright W$.

Another example: $\rm{Ad}:G \curvearrowright \mathfrak{g}$, and so on.

-
I prefer for the arrow to be a nearly-full circle, coming from X back to X. Warning: if you put the arrow on the lower part instead of upper, it looks like you've written G twice but in two different fonts. – Allen Knutson Oct 21 2010 at 2:31

I use the notation

$V \oplus^{\perp}W$

to denote orthogonal direct sum [Edit: direct sum of, say, subspaces of a given inner-product space].

Or

$(M,g) \times^{\perp} (N,g')$, or simply $M \times^{\perp} N$, to denote (orthogonal) cartesian product of Riemannian manifolds.

-

As a freshman, I "invented" the notation

$H \lhd ! \; G$

to say that $H$ is a characteristic subgroup of $G$, i.e. a subgroup invariant under any automorphism of $G$ (whereas a normal subgroup $N\lhd G$ is only invariant under the inner automorphisms).

-

If one needs to denote the fiber (not the stalk which is standardly denoted $\mathcal{F}_{x}$) of a sheaf $\mathcal{F}$ at the closed point $x$ of the $\Bbbk$-scheme $X$, one can write

$\mathcal{F}\mid_{x}$

After all, the fiber $\mathcal{F}\otimes_{\Bbbk}\;\kappa (x)$ is the restriction (pullback) of $\mathcal{F}$ to the point $x:\rm{Spec}\;\Bbbk\rightarrow X$.

The problem is that, when you identify vector bundles with locally free sheaves, the above notation clatches with the usual notation $E_x$ for the fiber of vector bundles. On the other hand almost always the context would be sufficient to clarify which of the two notations is being used.

-
In Ravi Vakil's thread on teaching schemes, there was some discussion of using the notation $\mathcal{F}(x)$, since given a section $f$ of the sheaf, its values as a function on $X$ are written $f(x)$ and lie in the fibers. – Ryan Reich Oct 21 2010 at 6:38
It's this answer, and the many, many comments below it: mathoverflow.net/questions/28496/…. Note that what I wrote is given by BCnrd about halfway down. – Ryan Reich Oct 21 2010 at 6:48
@Ryan: I appreciate the $\mathcal{F}(x)$ notation, but there's a (minor?) clutching with the widely used notation $\mathcal{O}_{X}(x)$ to denote the line bundle on the algebraic curve $X$ twisted by the divisor given by the point $x\in X$. – Qfwfq Oct 21 2010 at 11:44
It's true, this notation would lead to the unfortunate equality $\mathcal{O}_X(x) = \mathcal{O}_X/\mathcal{O}_X(-x)$. – Ryan Reich Oct 21 2010 at 16:22

I know some people absolutely DESPISE using coordinates and components to do "tensor analysis", but sometimes there is no recourse, and then Einstein's summation convention is a big help.

-
Have you ever seen Penrose's notation? It seems crazy at first, and I've never used it but it actually seems like a pretty good compromise between "abstract" notation and co-ordinates. – Deane Yang Oct 21 2010 at 1:55
show 1 more comment

I recently saw the following notation in the context of divisors on algebraic varieties, and I liked it very much.

Suppose that $D$ and $E$ are reduced divisors on a normal algebraic variety $X$. One can use $D \vee E$ to denote the reduced divisor with support equal to $D + E$ and $D \wedge E$ to denote $(D + E) - (D \vee E)$. I could imagine variants on this if $D$ and $E$ are non-reduced (involving taking max's, respecitvely mins, of the divisors component-wise).

EDIT: I'm slightly curious as to why this was downvoted. I guess it's too common to be interesting?

-
It is not me , I do not know algebraic variety, but as a guess may be there is some order there and so you have an inf and sup or even a lattice that would fully justify these notation and be rather common at that. – Jérôme JEAN-CHARLES Nov 23 2010 at 23:54

Since the standard notation for open interval $(a,b)$ can be confused with the coordinates, gcd, and other stuffs (open brackets have been used A LOT!), I've seem notations like

$]a,b[$

occurred in the book "Elementary Classical Analysis" by Marsden, and we can denote half-open half-closed interval like this:

$]a,b]$ or $[a,b[$.

-
I have to -1 this one. Nothing annoys me more than seeing $[x,y[ \cup ]a,b[$ and such. – Quadrescence Oct 21 2010 at 4:36
I was a fan of this notation until I read Quadrescence's comment. – Ryan Reich Oct 21 2010 at 6:30
The point is that $]a,b[$ is the French way to write the open interval (this explains Harry's comment). It is still taught in high school, and students never learn about $(a,b)$, even at university. Only researchers adapt to this notation once they write in English. I agree that $]a,b[$ is clearer. – Denis Serre Oct 21 2010 at 6:48
@Ryan Reich: Quadrescence's example is only annoying when improperly typeset. Look at the difference in LaTeX between $[x,y[\cup]a,b[$ [x,y[\cup]a,b[ versus $\left[x,y\right[\cup\left]a,b\right[$ \left[x,y\right[\cup\left]a,b\right[. – Willie Wong Oct 21 2010 at 17:22
I prefer Knuth's notation, which uses $(a.\,.b)$ for the open interval and $[a.\,.b]$ for the closed one. – Zsbán Ambrus Oct 22 2010 at 19:38

As mentioned in a comment, $\lfloor x\rfloor$ is much better notation than $[x]$ for denoting the greatest-integer function. Most especially since it doesn't collide with the $10^6$ other things that $[]$ is used for, e.g. the $0,1$ function Richard Borcherds mentioned.

I very much like, though haven't had much use for, the notation $n{q\atop \cdot}$ for $|GL_n(q)/B|$, pronounced "$n$ $q$-torial". Famously, it extends to a polynomial function of $q$, and when $q=1$ we have $n{1\atop \cdot} = n!$

(Oops: I left out the $/B$ the first time, thanks Jim and David.)

-
Allen, that's not quite right about the cardinality of $\mathrm{GL}_n(q)$. For instance, when $q=1$ you get $0$. (Maybe you want the number of points on $G/T$?) – James Borger Oct 21 2010 at 6:44
I'm pretty sure G/B is what Allen wants. – David Speyer Oct 22 2010 at 13:40

I like notation such as $2^X$ for the set of subsets of $X$ and ${X\choose k}$ for the set of $k$-element subsets. Also $[x^n]F(x)$ for the coefficient of $x^n$ in the power series $F(x)$, and multivariate notation like $x^\alpha$ for $x_1^{\alpha_1}\cdots x_n^{\alpha_n}$, where $x=(x_1,\dots,x_n)$ and $\alpha=(\alpha_1,\dots,\alpha_n)$.

-
I particularly like $X \choose k$ when X is say an algebraic variety and so $\chi {X \choose k} = {\chi(X) \choose k}$, where $\chi$ is the Euler characteristic... – Vivek Shende Oct 29 2010 at 2:29

Diagrammatic notation for tensors (Penrose diagrams, birdtracks, etc.). It makes many things like the invariance of tr(A B C) under cyclic permutation into empty statements.

-

Bourbaki dangerous bend symbol to mark dangerous or difficult ideas.

-
Haha. I didn't know this was due to Bourbaki, I learned it from Knuth's "The TeXbook". – Andres Caicedo Oct 21 2010 at 17:33
show 1 more comment
1. For rising and falling factorials: $x^{\overline{n}}$ and $x^{\underline{n}}$ à la Knuth. Much better than the traditional way to write the Pochhammer symbol: $(x)_n := x^{\overline{n}}$. In a book I'm writing, I use the notation $x^{\uparrow n}$ and $x^{\downarrow n}$, which I find much less clumsy (consider $(2x+1)^{\overline{6k-2}}$ vs $(2x+1)^{\uparrow6k-2}$). Anyway, the utility in either of these notations is seen in the umbral calculus; it makes the connection to "ordinary" calculus much more apparent, such as with $$\Delta x^{\uparrow n} = n x^{\uparrow n-1}\qquad\text{compared to}\qquad D x^n = nx^{n-1}.$$

2. The simple idea of omitting parentheses for function application: $f\,x$ as opposed to $f(x)$. I think this often makes some mathematics look cleaner, especially when the argument isn't especially complex. It also allows for some nice (= convenient) abuse of notation, such as in $$\left[ (-1)^{p - m - n} z \prod_{j = 1}^p \left( z D_z - a_j + 1 \right) - \prod_{j = 1}^q \left( z D_z - b_j \right) \right] G(z) = 0,$$ where $D_z:=d/dz$. Note this equation isn't a product (entirely); upon expansion, we'd have $D_z G(z)$ terms.

3. Do fractions count? Imagine having to write $$\sqrt{(x^2 + 2x + 1)\div (5x^3 - 3x^2 + 2x - 7)}$$ instead of $$\sqrt{\frac{x^2 + 2x + 1}{5x^3 - 3x^2 + 2x - 7}}.$$

4. Big-O notation. Though often abused, this is a much less clumsy way to express boundedness and asymptotics and errors and even lets you begin to do some algebra with them (provided you're careful). I don't think doing such is as obvious when you write it all out manually.

5. $\square(x)$ for the square wave, $\triangle(x)$ for the triangle wave, $Ш(x)$ for the Dirac comb (seriously, see Appel's "Mathematics for Physics and Physicists"). These are more cute than explicitly useful.

6. Notation used with musical isomorphisms as a way to do raising and lowering of indices. We have $X^\sharp$ which raises the index (in the context of Einstein summation) and $X^\flat$ which lowers the index. Here, $\flat$ and $\sharp$ are isomorphisms between tangent $TM$ and cotangent bundles $T^*M$: $\flat:TM\to T^*M$ and $\sharp:T^*M\to TM$.

7. Using $\operatorname{cis}\theta = \cos\theta + \mathrm{i}\sin\theta$ (cosine i sine), which is nice for obvious reasons (yes, $\omega = e^{\mathrm{i}\theta}$ is nice too) and $\operatorname{cas}\theta = \cos\theta + \sin\theta$ (cosine and sine), which is used in e.g., the Hartley transform.

8. Notations for hypergeometric functions $${}_pF_q \!\left( \left. \begin{matrix} a_1, \dots, a_p \\ b_1, \dots, b_q \end{matrix} \; \right| \, z \right) = {}_pF_q(\mathbf{a},\mathbf{b};z)$$ and Meijer-$G$ functions: $$G_{p,q}^{m,n} \!\left( \left. \begin{matrix} a_1, \dots, a_p \\ b_1, \dots, b_q \end{matrix} \; \right| \, z \right)=G_{p,q}^{m,n} \!\left( \left. \begin{matrix} \mathbf{a} \\ \mathbf{b} \end{matrix} \; \right| \, z \right)$$

9. Notation for general continued fractions: $$\underset{j=1}{\overset{\infty}{\LARGE\mathrm K}}\frac{a_j}{b_j}=\cfrac{a_1}{b_1+\cfrac{a_2}{b_2+\cfrac{a_3}{b_3+\ddots}}}.$$ The $\mathrm{K}$ comes from German's "Kettenbruch", which is "continued fraction."

I think that's good for now. There are probably lots more. :)

To end, I'll say one notation I do not like: the use of fraktur. Most of the time it just looks ugly and no one can actually write fraktur letters.

-
9. I am a fan of this. :D – J. M. Oct 21 2010 at 4:53
Knuth's <i> x to the n falling </i> is $x^{\underline{n}}$, not $x_{\underline{n}}$. – Bruce Arnold Oct 21 2010 at 13:50
I don't like the cis(θ) notation because it depends on the non-canonical choice of the square root of 1 (i.e., i). The exp(iθ) notation is better because the non-canonical choice is clearly visible. – Dmitri Pavlov Oct 21 2010 at 15:51
@KConrad: Sure you can, but you shouldn't. As with other scripts (say Latin or Cyrillic), the old script for writing German has a version for typesetting and a version for handwriting: Sütterlin (en.wikipedia.org/wiki/Sütterlin). If you like Fraktur, you will love it. – Maxime Bourrigan Oct 22 2010 at 15:00
@Bourrigan: I whole-heartedly disagree. Fraktur letters aren't that hard to write, and are easy to read. Sütterlin is illegible spaghetti, and even worse, the letters more often than not aren't what you expect them to be (the 'h' looks like an 'f', the 'e' like an 'n', etc.). It would be severely unpedagogical to use the script for anything other than to demonstrate how not to make a legible script. – Ketil Tveiten Nov 16 2010 at 9:26

The lack of a nice obviously symmetric notation for $\binom{a+b}{b}$ has bothered me; Dijkstra suggested in EWD 782 the notation $P(a,b)$, generalizing it also to $P(a_1,\ldots,a_k)$ for $\binom{a_1+\ldots+a_k}{a_1,\ldots,a_k}$. (Though I certainly disagree with him about $\binom{n}{k}$ being useless - you certainly do want to think about it that way a lot of the time.) I haven't actually had any reason to use this since I saw it but I can certainly think of times I would have.

Also the double-parentheses multichoose notation $\left(\!\binom{n}{k}\!\right)$ is nice because it lets you say "...and this is n multichoose k (which is equal to this binomial coefficient)" instead of just jumping directly to a binomial coefficient whose relevance may not be immediately obvious. But I suppose that's not really on the level of giving you a better way to look at things.

-
Isn't there a nice obviously symmetric notation for ${a+b} \choose b$, namely ${a+b} \choose {a,b}$? – Rasmus Oct 21 2010 at 18:34

Using $(a, b, ... )$ is handy to denote a column vector, which is the transpose of the row vector $[a, b, ... ]$, especially in linear text. Correspondingly, all displayed matrices should be written with brackets, not parentheses. This notation agrees with the usual identification of coordinates with column vectors.

-
I take it you also hate having to write the transpose when you're constrained to writing on one line? :) – J. M. Oct 21 2010 at 11:45
Yes, particularly when stacking vectors $a, b, ...$, in which case you have to write the artificial $[a^T, b^T, ... ]^T$ without this convention. – John Bentin Oct 21 2010 at 13:52
It is, of course, not $[a, b, \dots]^T$ but $[a\;b\;\dots]^T$. This is an awful convention because even once you remind yourself why the $T$ is there you (or I at least) am not convinced that it has any meaning other than to satisfy a badly chosen precedent. I don't know why anyone would vote this down. – Ryan Reich Oct 21 2010 at 16:27
Don't forget the invariant literature which uses $\left[v_1,...,v_n\right]$ not for the matrix formed by the columns $v_1$, ..., $v_n$, but for its determiannt... – darij grinberg Oct 21 2010 at 17:04

The notation $M^{\oplus n}$ and $M^{\otimes n}$ to denote, respectively, nth direct sum and nth tensor product.

The notation $X \mathbin{\pi} Y$ to denote product of objects in an abstract category, and the analogous with the "upside down $\pi$" for coproduct. I once have seen this being used by B.Keller in a talk. It'd be nice to have a smaller $\Pi$ (resp. $\amalg$) symbol instead.

-
$\sqcap$ resp. $\sqcup$? – Max Nov 8 2010 at 18:39
I prefer to use the same notation as in arithmetic: $\times$ for binary product, $\prod$ for product of an arbitrary family, $+$ for binary coproduct, $\sum$ of coproduct of an arbitrary family. A small pi-like symbol for binary product seems to me to be a step in the wrong direction. – Tom Leinster Dec 10 at 0:28
show 1 more comment

The notation for transversality:

$M \pitchfork N$

-

The ever-controversial reverse Polish notation for functions: $f(x) = xf$. Thus in composition, the order makes sense: $(g \circ f)(x) = x f g$ (this point is moot for the fortunate Hebrew- and Arabic-speaking mathematicians). I hate this notation in practice but I can't deny that it is objectively right and "just makes sense" in more or less the same way that the original post discusses writing $B^A = A \to B$. Please no one vote this up.

-
One partial compromise is to subscript a function $f: X \to Y$ as $f_{Y \leftarrow X}$. For instance, the composition of two inclusion maps $\iota_{Z \leftarrow Y} \circ \iota_{Y \leftarrow X}$ becomes $\iota_{Z \leftarrow X}$. – Terry Tao Oct 21 2010 at 17:11
Hehe, `objectively right'. – Greg Muller Oct 21 2010 at 18:42
@Terry Tao: I've seen that one from time to time, and it is pretty nice when you are dealing with different spaces (especially when you have a category of them and, as you write, a functorial assignment of maps). For self-maps of X, it leaves...something to be desired. – Ryan Reich Oct 21 2010 at 20:36
Abstractly an arrow from $X$ to $Y$ has and an orientation but no direction. It could be drawn either up/down/right/left/slant 30°/slant-45°/... . But we are heavy victims of the typographical habits, and this in spite of modern computer possibilities (not means :Tex being a terrible tool!). The best thing to do is to think of an arrow as slanted +200°or in three dimensions, then the typographical induction tends to disappear completely (at least for me and I guess for Hebrew/Arabic writers too). So the motto (not necessarily a categorist's one) is "let's do multidimensional algebra!". – Jérôme JEAN-CHARLES Nov 23 2010 at 23:50

I am fond of subscripting asymptotic notation with the parameters that the implied constant is allowed to depend on (and on the asymptotic parameter, if needed). e.g.

• $X = O_k(Y)$ (or $X \ll_k Y$, or $Y \gg_k X$) means that $|X| \leq C_k Y$ for some $C_k$ depending only on $k$.
• $X = o_{n \to \infty; k}(Y)$ means that $|X| \leq c_k(n) Y$ for some function $c_k(n)$ of both $k$ and $n$, which goes to zero as $n \to \infty$ for fixed $k$.
• (Rarer) $X = O_{n \to \infty; k}(Y)$ means that $|X| \geq C_k Y$ whenever $n \geq N_k$, for some $C_k$ and $N_k$ depending only on $k$.

Of course, if there is a parameter that influences all the constants (e.g. the ambient dimension) then it is better to explicitly state at the beginning that all constants will depend on this parameter so that one does not have to put in the explicit subscripts in all the time.

It can be instructive to rewrite some basic notions in analysis in this sort of notation, just to get a slightly different perspective. For instance, if $f: {\bf R} \to {\bf R}$ is a function, then:

• $f$ is continuous iff one has $f(y) = f(x) + o_{y \to x; f,x}(1)$ for all $x,y \in {\bf R}$
• $f$ is uniformly continuous iff one has $f(y) = f(x) + o_{|y-x| \to 0; f}(1)$ for all $x,y \in {\bf R}$
• A sequence $F = (f_n)_{n \in {\bf N}}$ of functions is equicontinuous if one has $f_n(y) = f_n(x) + o_{y \to x; F,x}(1)$ for all $x,y \in {\bf R}$ and $n \in {\bf N}$ (note that the implied constant depends on the family $F$, but not on the specific function $f_n$ or on the index $n$)
• A sequence $F = (f_n)_{n \in {\bf N}}$ of functions is uniformly equicontinuous if one has $f_n(y) = f_n(x) + o_{|y-x| \to 0; F}(1)$ for all $x,y \in {\bf R}$ and $n \in {\bf N}$
• $f$ is differentiable iff one has $f(y) = f(x) + (y-x) f'(x) + o_{y \to x; f,x}(|y-x|)$ for all $x,y \in {\bf R}$;
• (similarly for uniformly differentiable, equidifferentiable, etc.)

(These formulations are close to the nonstandard analysis formulations of these concepts, which uses similar but not quite identical asymptotic notation, but that is another story.)

-
The subscripts seem a little bulky. Couldn't you just write, e.g. $f(y) = f(x) + o_{y \to x; f,x}(1)$? I guess it's not really changing much. – Ryan Reich Oct 21 2010 at 17:18
Maybe it's time for you to go to really big-O notation: $X=\underset{n\to\infty}{\overset{k}{\LARGE\mathcal O}} Y$ for $O(\cdot)$ and then $f_n(y)=f_n(x)+\underset{|y-x|\to 0}{\overset{F}{\LARGE\mathrm O}} 1$. for $o(\cdot)$. :D – Quadrescence Oct 21 2010 at 17:34
The "really" big-O notation is a little bit confusing; since normally we write summation like this with $Y$ depends on the parameter $k$, but here we have the constant $C_k$ depends on it instead. – Hsien-Chih Chang Oct 22 2010 at 1:07
Chang: definite sums can depend on their upper index. – Quadrescence Oct 22 2010 at 6:08
I think O-o notation is one of the worse. It does not worth to improve --- better to start from scratch. – Anton Petrunin Jan 30 at 19:18
show 1 more comment

One can decorate a subscript or superscript by additional symbols to indicate what the subscript or superscript is doing. For instance, consider a truncation $f 1_{|f| \leq N}$ of a function to its values whose magnitude is at most $N$. One could of course call such a function something like $f_N$, but why not call it $f_{\leq N}$ instead? Then one can do things like "Decompose $f = f_{\leq N} + f_{>N}$, where $f_{\leq N} := f 1_{|f| \leq N}$ and $f_{>N} := f 1_{|f| > N}$." Notation of this type is sometimes used in PDE, particularly with regard to Littlewood-Paley frequency projections.

Similarly, one could imagine the operation of shifting $f$ by $N$ to be denoted something like $f_{+N}$ rather than $f_N$, etc..

-
Hum, I would prefer if $f 1_{|f| \leq N}$ be written as $f|_{\leq N}$ to make clear that it is a spatial cutoff. $f_{\leq N}$ suggest so me a vertical cut-off: $=f$ if $|f| \leq N$ and $=N$ otherwise. This could be a nice notation when doing Calderon-Zygmund... – Willie Wong Oct 21 2010 at 17:41
I'm just making this up, but it seems you could be more consistent with the projection $[x^n]$ as in $[x^n]f(x)$ by doing something like $[\le N]f$: "Decompose $f=[\le N]f + [>N]f$", or if you're daring, "decompose $f=([\le N] + [>N])f$." – Quadrescence Oct 21 2010 at 17:42