Historical (personal) examples of teaching-based research The phrase "teaching-based research" brings to mind research about teaching, though important, it is not what I mean. Unfortunately, I couldn't come up with a better phrase, thus please bear with me while I explain the intended meaning. 
I have taught multi-variable calculus several times. As usual of such repetition, I had the feeling that I know the concepts involved and how they are connected to each other and so on. But, when last week I was preparing for one of my sessions - in which I decided to use a bathymetric map (depth contours) rather than a topographic map (height contours) - a problem occurred to me for the first time. Imagining myself swimming to the shore while looking at the bathymetric map, it seemed "obvious" that if I wanted to take the shortest path the to shore (from where I was), moving in the opposite direction of the gradient would not be my choice! Prompted by my observation, I came to this quite "recent" paper "When Does Water Find the Shortest Path Downhill? The Geometry of Steepest Descent Curves" addressing the very same problem that whether gradient curves are geodesics. 
Now here is the question: Do you know any personal (or historical) examples of such "teaching-based research"? 
And, here is why I think the question is suitable for MO: 
Many mathematician friends of mine, for obvious reasons, prefer spending their time on research rather than teaching. Having a collection of such examples could be encouraging in particular for early career mathematicians. 
There is a recent movement to encourage "teaching inquiry" the point of which is  to "teach students to ask and explore mathematical questions". For that aim, it seems that  lecturers should be ready to be faced with some problems never posed before in the subjects that are too familiar to them, and better, be ready to pose such genuine questions in such contexts.          
Finally, it goes without saying that, it is a habit of mind to pose such questions in everyday research practice. It seems that what makes it difficult in teaching is rooted in an all-knowing feeling. If we know how to bypass such feeling, we could understand how  students might develop such a habit beyond procedural fluency and conceptual understanding.      
 A: My only paper on measure theory came out of an undergraduate lecture. I was explaining the Monotone Convergence Theorem and started to say "Of course, this goes completely wrong if you don't have monotone convergence from below, because ..." and then couldn't think of a satisfactory example. So I said "OK, I'll come back to you on that one!". 
The example I was looking for doesn't exist: if some non-negative measurable functions converge to another measurable function from below, it doesn't matter whether the convergence is actually monotone, the integrals still converge to the integral of the limit (whether the limit of the integrals is finite or not). In fact this is a two-line corollary of Fatou's Lemma. But I couldn't find this "Convergence from Below Theorem" in the literature, so I ended up publishing it!
See https://www.maths.nottingham.ac.uk/personal/jff/Papers/pdf/CBT.pdf
A: A personal experience: while teaching a freshman class in combinatorial game theory I had mentioned offhandedly some of what was had been determined about outcome classes of rectangular Domineering positions. Multiple students asked about the methods that were used to determine these outcome classes. In rereading the literature to prepare a follow-up lecture I realized that a number of new things could be said, which led to this paper.
A: According to David Fowler, (Dedekind's theorem: $\sqrt{2}\sqrt{3}=\sqrt{6}$,
Amer Math Monthly 99 (1992), 8, 725-733),
Dedekind constructed real numbers on Wednesday, November 24, 1858,
in the process of preparation to his first Introductory Calculus class.
A: The first time I taught forcing, I wanted to mention, as motivation, the fact that the independence of the continuum hypothesis (CH) or even of the axiom of constructibility (V=L) cannot be proved by the method of inner models. That fact was proved in Cohen's book, "Set Theory and the Continuum Hypothesis" under the assumption that there is a standard model (i.e., a transitive set model with standard $\in$) of ZF, and Cohen mentions that one could prove the same fact under the strictly weaker assumption that ZF is consistent. So I wanted to show my class the proof under this weaker hypothesis, but I couldn't figure out how to do the proof. That's a good thing, because, in fact, the conclusion doesn't follow from that weaker hypothesis. Discovering that and analyzing the situation a little further, I found that the fact in question is equivalent to the $\omega$-consistency of ZF, which is strictly stronger than consistency and strictly weaker than existence of a standard model. This work was too complicated to present in that class, but it became one of my early papers, "On the inadequacy of inner models" [J Symbolic Logic 37 (1972) 569-571].
A: Students' standard topological reasoning went often along this line: a set $S$ is not closed hence it's open (and vice versa). Thus I defined the topological student spaces, where each subset of a space is open or closed (or both). Actually, there was already an exercise in the standard Kelley's topology textbook where such spaces were called (a bit not precisely) the door spaces (a door is either opened or closed; however the door space name ignored the fact that a door cannot be both). The said exercise was restricted to the trivial case of Hausdorff spaces only.
Later, when A. Mishchenko visited Warsaw, he and I wrote a paper about the student spaces without using this name for technical reasons though--we have treated more general spaces hence the student spaces name didn't fit.
A: Perhaps the most important result about which it is documented that it came from teaching duties is the Stokes theorem. It was invented as an exam problem (pdf).
(Even more famous example, not from mathematics properly, but it eventually made a great influence on physics and mathematics, is the discovery of the Periodic Law by Mendeleev: he was thinking on the best order to survey the elements in his lectures).
According to Grothendieck, his famous work Esquisse d'un Programme was at least partially inspired by his teaching experience. Here is what he writes himself:

Les exigences d’un enseignement universitaire, s’adressant donc a des
  etudiants (y compris les etudiants dits “avances”) au bagage mathematique
  modeste (et souvent moins que modeste), m’ont amene a renouveler de
  facon draconienne les themes de reflexion a proposer a mes eleves, et de fil
  en aiguille et de plus en plus, a moi-meme egalement.

Here is the English translation (pdf), by Leila Schneps of a somewhat longer passage.
The remarkable feature of this example is that here we apparently deal not
just with teaching-based research but with teaching-based CHANGE OF THE SUBJECT of reaserch of one of the greatest mathematicians of 20th century:-)
According to a personal account of one of the authors, the main idea of this important paper:
MR2552110
E.Mukhin, V.Tarasov, A. Varchenko, The B. and M. Shapiro conjecture in real algebraic geometry and the Bethe ansatz, Ann. of Math. (2) 170 (2009), no. 2, pp 863–881, doi:10.4007/annals.2009.170.863 .
occured to him when he was teaching his undergraduate class and proving the theorem that
all eigenvalues of a Hermitean matrix are real.
And of course, I do not mention abundant examples when first rate research came out when a professor was trying to invent a thesis topic for his student...
EDIT. Complete English translation of the Esquisse d'un programme, is here (pdf).
A: Some years ago, I was teaching a course on nonparametric statistics and asked the students if some of them would be interested in a deeper investigation of the Pitman asymptotic relative efficiency between Pearson's, Kendall's, and Spearman's sample correlation coefficients. One of the students was interested, which led to the paper Berry--Esseen bounds for nonlinear statistics, as well as a number of further papers developing tools for Berry--Esseen bounds for nonlinear statistics and referred to there.   
A: Lawvere's introduction of ETCS (Elementary Theory of the Category of Sets) arose from his teaching undergraduate courses at Reed College. Quoting from the introduction of the TAC reprint: 

This elementary theory of the category of sets arose from a purely practical educational need. When I began teaching at Reed College in 1963, I was instructed that first-year analysis should emphasize foundations, with the usual formulas and applications of calculus being filled out in the second year. Since part of the difficulty in learning calculus stems from the rigid refusal of most textbooks to supply clear, explicit, statements of concepts and principles, I was very happy with the opportunity to oppose that unfortunate trend. Part of the summer of 1963 was devoted to designing a course based on the axiomatics of Zermelo-Fraenkel set theory (even though I had already before concluded that the category of categories is the best setting for “advanced” mathematics). But I soon realized that even an entire semester would not be adequate for explaining all the (for a beginner bizarre) membership-theoretic definitions and results, then translating them into operations usable in algebra and analysis, then using that framework to construct a
  basis for the material I planned to present in the second semester on metric spaces.
  However I found a way out of the ZF impasse and the able Reed students could indeed
  be led to take advantage of the second semester that I had planned. The way was to
  present in a couple of months an explicit axiomatic theory of the mathematical operations and concepts (composition, functionals, etc.) as actually needed in the development of the mathematics. Later, at the ETH in Zurich, I was able to further simplify the list of axioms. 

A: Hermann Grassmann's 1861 Lehrbuch der Arithmetik, the first book 
to recognise that induction is the foundation of number theory, was 
intended as a textbook for his high school students. It was so far
ahead of its time that the idea did not catch on until Dedekind and
Peano rediscovered the idea in 1888-1889.
A: While thinking about teaching finite fields for students who knew some group theory and what a field is, but had not seen abstract linear algebra, I wondered if there might be a way to show them a finite field has prime-power order by a method other than the standard one that relies on bases (i.e., the field has some characteristic $p > 0$, let $d$ be its dimension over $\mathbf F_p$, so the size of the field is $p^d$).  
I thought up the following argument: if $a$ and $b$ are nonzero elements in a finite field of characteristic $p$ then the function $f(x) = (b/a)x$ is an automorphism of the additive group of the field with $f(a) = b$, so $a$ and $b$ have the same additive order. Since $a$ and $b$ are arbitary, all nonzero elements have the same additive order. Then from $1$ having order $p$, every element has order $p$. (Or, if we don't want to mention characteristics of finite fields, let $p$ be some prime dividing the order of the finite field. By Cauchy's theorem there's an element of additive order $p$ and therefore all elements have additive order $p$.) A group where every nontrivial element the same prime order $p$ must have size equal to a power of $p$, since if the order of the group were divisible by another prime $q$ there'd be an element of order $q$ by Cauchy's theorem. When I later looked up E.H. Moore's 1893 paper on finite fields, he proves all nonzero elements share the additive order of $1$ and that this common order is a prime, but for the deduction that the order of the field is a power of that prime he uses linear algebra. (A link to Moore's paper is http://www.mathunion.org/ICM/ICM1893/Main/icm1893.0208.0242.ocr.pdf; see pp.213-215, noting he uses the word "mark" for what we'd call an element.)
Since this proof by group theory is largely unknown, I added it to the Wikipedia page on finite fields. I just noticed that it was removed in Feb. 2014 (http://en.wikipedia.org/w/index.php?title=Finite_field&diff=596682149&oldid=596659179) with the comment "deleted the longer proof because it is not something that one finds in a typical textbook." That a proof is not the usual one, while still not being outrageously tedious, seems like exactly the wrong reason to take it away; anyone can find the usual proof in textbooks. (Edit: I searched around and found this proof on math.stackexchange from Aug. 2014 at https://math.stackexchange.com/questions/72856/order-of-finite-fields-is-pn.)
A: I think there are quite a few of these.
A quote from J. V. Grabiner's Who Gave You the Epsilon?:

...there was another factor increasing interest in foundations, not only for Lagrange, but for many other mathematicians by the end of the eighteenth century: the need to teach. Teaching forces one's attention to basic questions. ... it was as a lecturer in analysis at the Ecole Polytechnique that Lagrange wrote his two major works on the calculus which treated foundations; similarly, it was 40 years earlier, teaching the calculus at the Military Academy at Turn, that Lagrange had first set out to work on the problem of foundations. Because teaching forces one to ask basic questions about the nature of the most important concepts, the change in the economic circumstances of mathematicians—the need to teach—provided a catalyst for the crystallization of the foundations of the calculus out of the historical and mathematical background. In fact, even well into the nineteenth century, much of foundations was born in the teaching situation; Weierstrass's foundations come from his lectures at Berlin; Dedekind first thought of the problem of continuity while teaching at Zurich; Dini and Landua turned to foundations while teaching analysis; and ... so did Cauchy. 

An excerpt from Dedekind's own description (from Stetigkeit und irrationale Zahlen; English translation in 'Essays on the Theory of Numbers' and Ewald's 'From Kant to Hilbert' and probably elsewhere):

As professor in the Polytechnic School in Zürich I found myself for the first time obliged to lecture upon the ideas of the differential calculus and felt more keenly than ever before the lack of a really scientific foundation for arithmetic. In discussing the notion of the approach of a variable magnitude to a fixed limiting value, and especially in proving the theorem that every magnitude which grows continuously but not beyond all limits, must certainly approach a limiting value, I had recourse to geometric evidences. ... This feeling of dissatisfaction was so overpowering that I made the fixed resolve to keep mediating on the question till I should find a purely arithmetic and perfectly rigorous foundation for the principles of infinitesimal analysis.

A quote from T. W. Körner's The Naked Lecturer:

For most mathematicians lecturing is part of their job. A few mathematicians grudge every moment of teaching as a moment taken from their research. Of course, teaching may occasionally aid research. Conway lectured on the construction of the real numbers starting with naive set theory, giving a different version of the standard constructions each year. I suspect that he would not have produced the theory of surreal numbers if he had not given those lectures.

A: I have at least 2 papers that came directly out of teaching a class.  


*

*The first came from a graduate (but introductory) number theory class.  One always says the class group of a number field measures the failure of unique factorization, which is in some sense clear from the definition, but it is not obvious how it precisely measures the failure of unique factorization.  Specifically, once you know the class group, how can you tell how many distinct factorizations an element of the ring of integers has?  I tried working out some examples, and 2-3 months later I wrote this paper which answers this question.  Fortunately, I got the results in time to explain in my class and included them in my course notes (Ch 5).

*The second paper came out of an honors calculus class.  I was explaining asymptotics at infinity, and as an "honors bonus" talked about rates of growth, giving examples such as the $O(\log n)$ binary search and the $O(n)$ linear search.  Corollary: you should always keep your room clean to minimize your search time.  Then I thought about what if you need to account for cleanup cost, and did some numerical calculations to tell them if you have $\le 12$ objects in your room, you should never clean it up.  I tried to work this out completely by the end of the semester to tell them a complete answer (under a given model), but it was trickier than I thought, and I worked on this with a colleague, which years later resulted in this paper.  (Note: my original numerical simulations did not exactly follow the main model we used in that paper, so the $\le 12$ is $\le 8$ in the paper.)
A: The question I posed here
Can one exhibit an explicit Kuratowski infinite set without invoking Replacement?
came straight out of my experience teaching undergraduate set theory. For various reasons, I prefer to postpone Replacement. This pedagogical decision renders the usual formulation of the Axiom of Infinity (inductive sets and von Neumann naturals) stronger than necessary, in an arbitrary way. Were I to use Zermelo naturals, that would be stronger in a slightly different arbitrary way. 
After several years, I eventually engaged the issue, and in fact proved what is suggested in the posting, namely that no specific infinite set can be exhibited in Zermelo set theory where "There exists an infinite set" is taken as the Axiom of Infinity. With more work (still in progress) I believe I can even prove this in the stronger system Zermelo (with the above infinity axiom) + Transitive Containment.
A: Stein's method, which has grown to be a key tool in probability theory, was developed to give a new proof of Hoeffding's combinatorial central limit for a statistics lecture. See Wikipedia and linked articles for more.
A: There are lots of works and research about proposing modern and more dynamic research-teaching approach to students. Unfortunately, they are principally jailed in didactic team research and journals, of quite restrained and confidential diffusion. Moreover, the mathematical domains and tools used in these studies are often elementary school geometrics or combinatorics, around very specific problems.
If I should propose a pleasant reading about how to teach mathematics with an approach promoting research, initiative, curiosity and criticism about objects, proofs, and statements, it is I. Lakatos, Proofs and Refutations. The Logic of Mathematical Discovery.
The book relates a class debate on the proof of Euler's formula for the characteristic on graphs. The very many viewpoints leads to criticism : from the original intuition and observations leading to the conjecture, definitions have to be precised, proof standards discussed, legal tools chosen, etc.
It is a nice and well-presented reflexion about research, mathematical truth, importance of trying and failing, etc. 
