Has incorrect notation ever led to a mistaken proof?

Question

In mathematics we introduce many different kinds of notation, and sometimes even a single object or construction can be represented by many different notations. To take two very different examples, the derivative of a function $y = f(x)$ can be written $f'(x)$, $D_x f$, or $\frac{dy}{dx}$; while composition of morphisms in a monoidal category can be represented in traditional linear style, linearly but in diagrammatic order, using pasting diagrams, using string diagrams, or using linear logic / type theory. Each notation has advantages and disadvantages, including clarity, conciseness, ease of use for calculation, and so on; but even more basic than these, a notation ought to be correct, in that every valid instance of it actually denotes something, and that the syntactic manipulations permitted on the notation similarly correspond to equalities or operations on the objects denoted.

Mathematicians who introduce and use a notation do not usually study the notation formally or prove that it is correct. But although this task is trivial to the point of vacuity for simple notations, for more complicated notations it becomes a substantial undertaking, and in many cases has never actually been completed. For instance, in Joyal-Street The geometry of tensor calculus it took some substantial work to prove the correctness of string diagrams for monoidal categories, while the analogous string diagrams used for many other variants of monoidal categories have, in many cases, never been proven correct in the same way. Similarly, the correctness of the "Calculus of Constructions" dependent type theory as a notation for a kind of "contextual category" took a lot of work for Streicher to prove in his book Semantics of type theory, and most other dependent type theories have not been analogously shown to be correct as notations for category theory.

My question is, among all these notations which have never been formally proven correct, has any of them actually turned out to be wrong and led to mathematical mistakes?

This may be an ambiguous question, so let me try to clarify a bit what I'm looking for and what I'm not looking for (and of course I reserve the right to clarify further in response to comments).

Firstly, I'm only interested in cases where the underlying mathematics was precisely defined and correct, from a modern perspective, with the mistake only lying in an incorrect notation or an incorrect use of that notation. So, for instance, mistakes made by early pioneers in calculus due to an imprecise notion of "infinitesimal" obeying (what we would now regard as) ill-defined rules don't count; there the issue was with the mathematics, not (just) the notation.

Secondly, I'm only interested in cases where the mistake was made and at least temporarily believed publically by professional (or serious amateur) mathematician(s). Blog posts and arxiv preprints count, but not private conversations on a blackboard, and not mistakes made by students.

An example of the sort of thing I'm looking for, but which (probably) doesn't satisfy this last criterion, is the following derivation of an incorrect "chain rule for the second derivative" using differentials. First here is a correct derivation of the correct chain rule for the first derivative, based on the derivative notation $\frac{dy}{dx} = f'(x)$:

$$\begin{align} z &= g(y)\\ y &= f(x)\\ dy &= f'(x) dx\\ dz &= g'(y) dy\\ &= g'(f(x)) f'(x) dx \end{align}$$

And here is the incorrect one, based on the second derivative notation $\frac{d^2y}{dx^2} = f''(x)$:

$$\begin{align} d^2y &= f''(x) dx^2\\ dy^2 &= (f'(x) dx)^2 = (f'(x))^2 dx^2\\ d^2z &= g''(y) dy^2\\ &= g''(f(x)) (f'(x))^2 dx^2 \end{align}$$

(The correct second derivative of $g\circ f$ is $g''(f(x)) (f'(x))^2 + g'(f(x)) f''(x)$.) The problem is that the second derivative notation $\frac{d^2y}{dx^2}$ cannot be taken seriously as a "fraction" in the same way that $\frac{dy}{dx}$ can, so the manipulations that it justifies are incorrect. However, I'm not aware of this mistake ever being made and believed in public by a serious mathematician who understood the precise meaning of derivatives, in a modern sense, but was only led astray by the notation.

Edit 10 Aug 2018: This question has attracted some interesting answers, but none of them is quite what I'm looking for (though Joel's comes the closest), so let me clarify further. By "a notation" I mean a systematic collection of valid syntax and rules for manipulating that syntax. It doesn't have to be completely formalized, but it should apply to many different examples in the same way, and be understood by multiple mathematicians -- e.g. one person writing $e$ to mean two different numbers in the same paper doesn't count. String diagrams and categorical type theory are the real sort of examples I have in mind; my non-example of differentials is borderline, but could in theory be elaborated into a system of syntaxes for "differential objects" that can be quotiented, differentiated, multiplied, etc. And by saying that a notation is incorrect, I mean that the "understood" way to interpret the syntax as mathematical objects is not actually well-defined in general, or that the rules for manipulating the syntax don't correspond to the way those objects actually behave. For instance, if it turned out that string diagrams for some kind of monoidal category were not actually invariant under deformations, that would be an example of an incorrect notation.

It might help if I explain a bit more about why I'm asking. I'm looking for arguments for or against the claim that it's important to formalize notations like this and prove that they are correct. If notations sometimes turn out to be wrong, then that's a good argument that we should make sure they're right! But oppositely, if in practice mathematicians have good enough intuitions when choosing notations that they never turn out to be wrong, then that's some kind of argument that it's not as important to formalize them.

Answer 1

Here is an example from set theory.

Set theorists commonly study not only the theory $\newcommand\ZFC{\text{ZFC}}\ZFC$ and its models, but also various fragments of this theory, such as the theory often denoted $\ZFC-{\rm P}$ or simply $\ZFC^-$, which does not include the power set axiom. One can find numerous instances in the literature where authors simply define $\ZFC-{\rm P}$ or $\ZFC^-$ as "$\ZFC$ without the power set axiom."

The notation itself suggests the idea that one is subtracting the axiom from the theory, and for this reason, I find it to be instance of incorrect notation, in the sense of the question. The problem, you see, is that the process of removing axioms from a theory is not well defined, since different axiomizations of the same theory may no longer be equivalent when one drops a common axiom.

And indeed, that is exactly the situation with $\ZFC^-$, which was eventually realized. Namely, the theory $\ZFC$ can be equivalently axiomatized using either the replacement axiom or the collection axiom plus separation, and these different approaches to the axiomatization are quite commonly found in practice. But Zarach proved that without the power set axiom, replacement and collection are no longer equivalent.

• Zarach, Andrzej M., Replacement $\nrightarrow$ collection, Hájek, Petr (ed.), Gödel ’96. Logical foundations of mathematics, computer science and physics -- Kurt Gödel’s legacy. Proceedings of a conference, Brno, Czech Republic, August 1996. Berlin: Springer-Verlag. Lect. Notes Log. 6, 307-322 (1996). ZBL0854.03047.

He also proved that various equivalent formulations of the axiom of choice are no longer equivalent without the power set axiom. For example, the well-order principle is strictly stronger than the choice set principle over $\text{ZF}^-$.

My co-authors and I discuss this at length and extend the analysis further in:

• Gitman, Victoria; Hamkins, Joel David; Johnstone, Thomas A., What is the theory ZFC without power set?, Math. Log. Q. 62, No. 4-5, 391-406 (2016), DOI:10.1002/malq.201500019, ZBL1375.03059.

We found particular instances in the previous literature where researchers, including some prominent researchers (and also some of our own prior published work), described their theory in a way that leads actually to the wrong version of the theory. (Nevertheless, all these instances were easily fixable, simply by defining the theory correctly, or by verifying collection rather than merely replacement; so in this sense, it was ultimately no cause for worry.)

Answer 2

This might not quite count, but if you start with a principal $G$-bundle $f:P\rightarrow B$, there are two natural ways to put a $G\times G$ structure on the bundle $P\times G\rightarrow B$ given by $(p,b)\mapsto f(p)$. Because it is standard notational practice to denote such a bundle by simply writing down the map $P\times G\rightarrow B$, there is nothing in the notation to distinguish between these structures, and therefore the notation leads you to believe they're the same.

By following this lead, Ethan Akin "proves" in the 1978 JPAA paper $K$-theory doesn't exist (https://doi.org/10.1016/0022-4049(78)90032-4) that the $K$-theory of $B$ is trivial, for any base space $B$. He reports that it took three Princeton graduate students (including himself) some non-trivial effort before they found the error.

This might meet the letter of your criterion by virtue of having made it into print, but probably violates the spirit because the author had already discovered the error, and indeed the whole point of the paper was to call attention to it.

Answer 3

This will probably not be considered a serious mistake, but maybe it counts:

According to Dray, Manogue if you ask the following question to scientist:

Suppose the temperature on a rectangular slab of metal is given by $T(x,y)=k(x^2+y^2)$ where $k$ is a constant.

What is $T(r,\theta)$?

A: $T(r,\theta)=kr^2$
B: $T(r,\theta)=k(r^2+\theta^2)$
C: Neither

most mathematicians choose B while most other scientists choose A.

(I don't know if this experiment was ever done on a large scale. I do know some people who studied mathematics and have tried to argue that A is the right answer.)

This question is called Corinne's Shibolleth in this article of Redish and Kuo, where it is discussed further.

Answer 4

Ramanujan's notebooks are an interesting case study. As discussed in detail in Chapter 24 ("Ramanujan's Theory of Prime Numbers") of Volume IV of Bruce Berndt's series Ramanujan's Notebooks, Ramanujan made a number of errors in his study of $\pi(x)$, the number of primes less than or equal to $x$. It is hard to say for sure that these errors are specifically due to Ramanujan's notation rather than some other misconception, but I think a case can be made that his notation was a contributing factor. For example, Berndt writes:

It is not clear from the notebooks how accurate Ramanujan thought his approximations $R(x)$ and $G(x)$ to $\pi(x)$ were. (Ramanujan always used equality signs in instances where we would use the signs $\approx$, $\sim$, or $\cong$.) According to Hardy, Ramanujan, in fact, claimed that, as $x$ tends to $\infty$, $$\pi(x)-R(x) = O(1) = \pi(x) - G(x),$$ both of which are false.

One could therefore argue that Ramanujan's careless use of equality signs contributed to his overestimating the accuracy of his approximations. On the other hand, one could also argue that Ramanujan's mistake was more fundamental, traceable to his inadequate understanding of the complex zeros of the zeta function.

Ramanujan also used (in effect) the notation $d\pi(x)/dx$, and one could argue that some of his misunderstandings were traceable to not having a proper definition of the notation $d\pi(x)/dx$ and yet assuming that it denoted a definite mathematical object with specific properties. Ramanujan was aware of the need for some justification because Hardy voiced his objections, and attempted to defend his notation (in this context, $n=\pi(x)$):

I think I am correct in using $dn/dx$ which is not the differential coefficient of a discontinuous function but the differential coefficient of an average continuous function passing fairly (though not exactly) through the isolated points. I have used $dn/dx$ in finding the number of numbers of the form $2^p3^q$, $2^p+3^q$, etc., less than $x$ and have got correct results.

However, as Berndt explains, Ramanujan's defense is inadequate. For more discussion, I recommend reading the entire chapter.

Answer 5

Edward Nelson’s proof of the inconsistency of Peano arithmetic (and weaker systems) had an error which he saw only after Terence Tao refined some notation. Specifically, Tao reformulated Chaitin’s theorem from

Given a theory $T$, there exists an $\ell$ with the property that, if $T$ is consistent, then there does not exist an $x$ such that $T$ can prove $K(x)>\ell$

to

Given a theory $T$, there exists an $\ell(T)$ with the property that, if $T$ is consistent, then there does not exist an $x$ such that $T$ can prove $K(x)>\ell(T)$

In particular, this allowed Tao to note that $\ell(T) < \ell(T’)$ is possible even when $T’$ is a restricted version of $T$, contrary to an implicit assumption in Nelson’s proof.

Answer 6

I am aware of a few articles discussing certain Macdonald polynomials in the introduction (and motivation) in the introduction, and then proceed to study properties of another family of polynomials.

The particular polynomials that are studied are not the same ones as in the Macdonald polynomials in the introduction (only similar), but the exact same notation/symbol is used for both these two families of polynomials.

Answer 7

In module theory, there is a choice of which side the scalars acts on. Then, there is also the choice of which side the endomorphisms of the module act on.

Let $M$ be a right module, with scalars coming from a ring $k$, and let $E={\rm End}(M_k)$. If we let $E$ act on the opposite side as $k$, so on the left, then we have a very nice associativity-like compatibility of actions, given by $$e(m\alpha)=(em)\alpha,$$ for any $e\in E$, $m\in M$, and $\alpha\in k$.

On the other hand, if we have $k$ and $E$ act on the same side, then the compatibility rule for the actions is ugly and harder to keep track of. I've seen wrong proofs because authors have endomorphisms act on the same side as scalars, but incorrectly remember the weird compatibility/multiplication rules.

Something similar happens with group actions, where if the action happens on the "wrong" side, then some inverses have to be added, and a natural associativity kind of rule instead has the order of the elements all messed up. (I've talked to some group theorists who were not even aware that their ugly group action rules could be fixed by acting on the other side.)

There are a couple historical reasons for this. First, we are used to function composition on the left, but also like working with left modules. So, having endos on the right leads to a right composition rule.

Second, many people start learning about modules using scalars from a commutative ring (e.g., vector spaces). This sometimes leads to problems as people get too comfortable moving scalars around willy nilly, even though this only works (generally) for commutative rings. Generally, we can view any right $R$-module $M_R$ as a left $R^{\rm op}$-module. Some authors are simply unaware of this "opposite ring" issue, since it magically disappears when working with commutative rings. I've seen some authors make errors in proofs by claiming some sort of simultaneous left and right module structure over a general ring $R$, when in fact it only works if $R$ is commutative. Also, one has to be careful to not confuse the endo ring of $M_k$ with the endo ring of $_kM$ (even when $k$ is commutative)! This is where I think this answers the OP, because the "understood" way to manipulate the syntax (of just moving scalars to the other side) is a misunderstanding of what is really happening (both moving and "oppositivizing").

In summary: Any module notation where one does not carefully keep track of which side scalars and endomorphisms act, and which doesn't put endos on the opposite side, inadvertantly hides naturality and often leads to hard-to-remember multiplication rules, making errors in proofs much easier.

Answer 8

I'm not entirely sure what the difference between wrong notation and wrong "underlying mathematics" is, so I'm going to present a few different examples, and hopefully this clarifies the question. (Perhaps one of these examples provides an answer to the OP; maybe the one about set collection?)

I. Incorrect transfer of notation

The notion of prime factorization was studied by the early Pythagoreans and Euclid in his Elements. Over time, a notation/terminology for this behavior was built up. So if someone wrote “Let $a=p_1p_2 \dotsm p_k$ be the prime factorization of $a$.” then it was clear what was meant. This factorization was unique, up to order, and up to unit multiples, and couldn't be extended further.

When more general rings of integers began to be studied, the notation of prime factorization was transferred to this new setting. In that context, an incorrect proof of Fermat's Last Theorem was given. Why was it incorrect? Because the notation only applies to UFDs.

This specific incorrect use of notation has had ripple effects in the field of abstract algebra, and has led many algebraists to be very careful when assuming behaviors under different assumptions. We don't want to repeat that fundamental mistake.

In modern mathematics we see the effect being that many statements have many very precise hypotheses.

Other examples of "incorrect transfer" abound. As a noncommutative ring theorist, I see this all the time, when looking at transfers from the commutative setting.

II. Imprecise reference to modeled objects

Nowadays we take the notation "Let $f$ be a function" for granted. But it actually took a while for mathematicians to formalize exactly what a function is (and for that formalization to be almost universally accepted). A function could have been all sorts of different things, including Dirac's delta "function".

I also view Joel's answer, of ZFC-P, to be of the "imprecise reference" type. The original understanding of that notation was not precise enough to ground it in a specific theory.

III. Inconsistent notation

Some notation refers to nonexistent objects. For instance, in set theory Kunen's inconsistency theorem tells us (among other things) that there is no nontrivial elementary embedding $j\colon V\to V$. The notation $j$ I just introduced refers to no actual object.

This is a feature of mathematics, not a bug. It is common in many branches of mathematics to introduce notations for things that later turn out to not exist at all. There is nothing being modeled by the notation, and yet the notation is a useful nonsense!

One of the famous examples of an inconsistent notation, that was initially believed by prominent mathematicians to be consistent, was that of arbitrary collection. As Russell and others noticed, the collection $\{x\, :\, x\notin x\}$ of all sets that aren't members of themselves seems innocuous enough, but it is self-contradictory.

IV. Misleading or overused notation

One of the easiest traps to fall into, regarding notation, is when that notation itself is misleading.

For instance, mathematicians often overuse the equals sign "=", having it mean different things in different contexts. Occasionally, it is even used for a non-symmetric relation symbol, such as in $x=O(x)$, even though the symbol "=" itself is visually symmetric. We are trained to treat symmetrically written relation symbols as implicitly symmetric.

In a separate answer, I mentioned how modules over commutative rings possess a similar issue. If $k$ is a commutative ring, and $M$ is a module over that ring, then we want to say $_k M=M_k$, because either module structure gives us the "same" information. Yet, a left module is different than a right module, and their endo rings are different, in meaningful ways.

Matt F.'s answer also seems to fall in this category. It is dangerous to remove parameters from a notation.

Per Alexandersson's answer is another instance of this phenomenon. Using the same notation in the same article for two different types of objects is misleading.

V. Unformalizable notation

Some errors in proofs come from incorrect uses of language. (Here, I'm treating language itself as a notation for the mathematical concepts under study.) Some concepts we think are well-defined, since we use them in our everyday conversations, are in fact not. This is brought home very well with some of the classic liar paradoxes, such as "This sentence is false." Typically, many of these paradoxes are resolved by noting a failure to recognize the "use/mention distinction" in language.