First, I don't like using the term "Museum", which has too many undesirable implications for me. I have to say I like the word "Factory".
Second, it seems to me that most exhibits give only an impressionistic, usually visual view-from-the-outside of mathematics. For me mathematics is a powerful tool combining deductive logic and abstraction, and I'd like to see exhibits or "labs", where ordinary people are allowed to experience the power of mathematics firsthand by showing them how to use deductive logic and abstraction themselves to gain new knowledge or insight. This, of course, means making the visitor work or think harder than usual, but I think it would be well worth having some exhibits like this, because I think it would create a deeper level of both understanding and excitement about mathematics.
I can't claim to have many concrete examples to offer, but one that comes from my experience teaching precalculus and calculus is to have an exhibit that introduces people to what a function is and then showing them in very concrete terms what a derivative is (i.e., the sensitivity of the output to changes in the input) and also the definite integral (if the function is measuring a rate of change then the definite integral recovers the total or net change). The important here is avoid an exhibit that just shows this visually but to actually make visitors work through a series of exercises (almost as if they were calculus students themselves) where they learn through firsthand experience. The analogy for me is sports or crafts (like, say, knitting). Instead of having visitors just watch someone else do things or look at the finished product, let them actually have the experience of doing the craft of mathematics (I like thinking of math as a craft rather than a science or body of knowledge or whatever) themselves.
