Rather than providing concrete examples, I would like to make a suggestion about two possible guiding lines.
If you agree that mathematics is about solving problems of a certain kind, then two sensible goals for a museum of mathematics could be showing:
- what kind of problems mathematics deals with, and
- how it deals with them.
The first goal cannot be exhaustive for obvious reasons, but I think it should be broad enough to give the visitor an idea of the diversity of mathematics (applied or not), i. e. examples from as many various branches as possible should be given.
For the second goal, Polya's views about how to solve mathematical problems could be helpful, or modern works about "visual" mathematical thinking. I think visitors should be able to feel some connection between mathematical problem solving and applying "common sense" strategies (here is a hidden goal: to demistify a bit the work of mathematicians).
To focus on one or the other goal in different degree can result in very different kinds of museum, but I think any of them would deserve the name "Museum of mathematics". Not so one that didn't meet any of both goals.
