How to Discover Counterexamples and Required Objects [closed]

What are strategies or tips, which research mathematicians have discovered through their work and experience, that would help undergraduates learn how to discover counterexamples or find an object on their own? Here, I define an object to be a bijection or function or or matrix or ... that satisfies the demanded properties. Although an undergraduate, I am posting here because I am interested in opinions and perspectives from research mathematicians. My question is motivated by the following:

1. Lecturers and textbooks almost never illustrate or teach how to construct counterexamples and requested objects. They start immediately with a claim, without any explanation or even summary of the claim's origins, and then check the definition(s). Yet I find the hard part to be the rough work and thought processes that produced the counterexample.

2. Exams and problems have always required me to discover counterexamples or objects. So my question seems to constitute fundamental skills that are significantly assessed and graded, but they are rarely taught, if at all

I realise that counterexamples and examples may require years to conceive and that experimentation is crucial. So the rough work and thought processes might be muddled. Nonetheless, what are some steps towards increased productivity? For example, exam conditions encumber "playing around" for a counterexample or object.

-

closed as too broad by Andy Putman, Alexandre Eremenko, Cam McLeman, Ramiro de la Vega, Eric WofseyJul 3 '13 at 21:34

There are either too many possible answers, or good answers would be too long for this format. Please add details to narrow the answer set or to isolate an issue that can be answered in a few paragraphs.If this question can be reworded to fit the rules in the help center, please edit the question.

This seems to me a better question for math.SE (I don't think it's aimed at research mathematicians as stated). Voting to migrate it there. –  Daniel Moskovich Jul 3 '13 at 4:55
I'm not so sure this is off-topic, although perhaps it is "too broad". Anyone else here fond of, or familiar with, the exploits of Korner, or Read? –  Yemon Choi Jul 3 '13 at 5:43
I think questions about teaching university-level math have been entertained on MO in the past, and this one asks how to teach a skill valuable in mathematical research. So it's not at all clear to me that this is off-topic. –  Gerry Myerson Jul 3 '13 at 7:21
Even restricted to the undergraduate level, finding mathematical objects with specified properties is an awfully broad topic for a MathOverflow question. Plus the level really matters (one might give rather different advice to a beginning undergrad, a more experienced undergrad, a beginning grad student, or a postdoc), and aiming this at undergrads makes it less of a good fit for MO. –  Henry Cohn Jul 3 '13 at 12:14
It is not true that "Lecturers and textbooks almost never illustrate or teach how to construct counterexamples"; it may be true in lower level courses where worst case analysis is not done due to audience limitations, but starting with upper level undergraduate courses counterexamples become essential, e.g. a point set topology course is mainly about counterexamples. –  Igor Belegradek Jul 3 '13 at 19:50

In research, one should not believe too strongly one way or the other. Trying to construct a counterexample, and trying to prove that there aren't any, are two ways of working with the same problem. One should try both ways instead of choosing. If you fail to prove a theorem, you might still obtain some constraints on a potential counterexample, thereby narrowing down the search. Conversely, if you fail to find a counterexample, you might get some hints to why there aren't any.

-
I've used this strategy to answer this -mathoverflow.net/questions/131930/… - so it works sometimes, at least. –  François G. Dorais Jul 4 '13 at 3:17

Search for extreme cases. In geometry, you may test your conjecture on an obtuse triangle or on triangles with a very small angle and see whether it is true in this case - otherwise you have found a counterexample. If it is a conjecture about a function on the reals, try it for large values.

Search for simple cases. In topology, the trivial topology $\{ \emptyset, S \}$ on a set $S$ may show some unexpected behaviour that you had not in mind when creating your conjecture: it is e.g. not Hausdorff. For matrices, a counterexample may be found already among the $2 \times 2$ matrices, as you can see here. (The whole discussion for which this is an answer is of course worth studying.)

It seems that for every theory there is a collection of objects that are expecially un-intuitive - so if you work within a certain theory you can build you own collection of mathematical objects that behave "wrong" and on which you can test your conjectures.

-
While I agree with your main point, I feel the need to nit-pick a little: I've never heard the "chaotic" topology to refer to the discrete topology that you cite. On the other hand, I have heard it used to describe the indiscrete topology $\{\emptyset, S\}$... –  Simon Rose Jul 3 '13 at 19:45
@SimonRose (Markus Redeker) I assume the two extremal toplogies are flipped relative to the text, since otherwise not only the name, but also and perhaps more importantly the assertion that the one of all sets is not Hausdorff appears wrong. –  quid Jul 3 '13 at 20:45
You are both right. I have corrected the terminology. –  rem Jul 4 '13 at 6:46
You say I am right, but you did not address the problem I raise in your edit. The topology consisting of all sets is not "not Hausdorff" contrary to what you (still) write. –  quid Jul 4 '13 at 14:39
@quid Many thanks, I corrected that too. I also removed the reference to the discrete topology, since I cannot think now of good counterexamples derived from it. (Others might find some.) –  rem Jul 6 '13 at 7:12

In general I believe in „practice makes perfect”. Thus, books like Counterexamples in Calculus by S. Klymchuk could be very useful.

-