A moving sculpture approximating Smale's turning of the sphere inside out. (but what material would you use?)
A sphere made out of elastic plastic with fotoreactive proteins. The proteins are laid down so that they only react when antipodal points touch.
A similar thing for Brower's theorem instead of Borusk-Ulam, with two discs.
A sphere made out of some flexible but not so flexible material, the spectator gets to sculpt the sphere into some shape, then he plays the shape with a stick. There are (say) seven choices of sticks in front of him each corresponding to one eigenvalue of the shape. When the visitor plays the sphere a computer computes the corresponding eigenvalue and translates it into a sound (maybe the sound of a drum).
A family of itouch-made-material 2d-surfaces, and a big sphere in the middle of the room. When one traces a curve (with the finger) on the surfaces. The Gauss map on the big sphere in the middle is animated.
A linkage that has the earth at the center of the solar system and traces the moving planets perfectly (this can be done by universality of Thurston and Kapovich-Millson right?) The spectator stands in the middle and sees the planets move around him. At the end he gets to wear an Inquisition hat and burn a paper sculpture of Galileo.
A metalic model to see percolation: The vertices will be represented by magnetized vertical thin tubes coming from the floor. The spectator stands in the second floor and from above (not above the spectator but above the magnetized tubes) a bunch of small tubes (edges) fall. Depending on the strength of the magnets (controlled by the user) some stay sticking to the tubes and some go to the ground, the spectator is asked to repeat the experiment many times and conjecture with what probability this random graph percolates.
In a dark room. A hospital bed with a set of cards is lighten like in a noir film. The visitor is supposed to play solitaire lying on the hospital bed, if he gets to the end in one round a screening of the H-bomb appears on a big screen in front of him. The sound of the bomb is heard in very loud speakers so that everybody in the museum hears this.
A 3d animation of a contorted 2-sphere Ricci flowing to a round sphere. Again the visitor gets to choose the starting sphere. At the end he is given a phone number. He tries to call Perleman.
A huge fountain that doesn't work.
An observatory with stars at random positions in which suddenly log n of them turn out to form a convex polygon. It should look like an astrological map.
Many microscopes looking at cells growing. In the first one like f(t)=t, then f(t)=t^2 and a few polynomials more. Then f(t)=2^t. (Can this be done with unlimited resources? I'm not joking, this is a honest experimental biology question.)
A mechanically transformable translucent skate park. Each configuration of the skate park corresponds to a link, an element in pi_1(C_n(R^3)). A tube of say two meters of radious traces the curve followed by one of the points in configuration space. So there are say 5 base points and 40=10*4 switches (like in a wiring diagram but going around 5 circles) The user gets to select what switches (adjecent transpositions are on).
The skaters are encouraged to use a helmet.
A liquid based model explaining Kepler's laws with a very eccentric ellipse.
A finance millionaire looking at the numbers of the stock market, very focused.
Three hallways that meet at the center. One with triangles on the walls, other with curves of functions, the third one with polynomials. At the center (e^i\pi=-1). Like that is a bit cheesy but if there was a computer app illustrating the geometry of elementary operations (+ and *) that foced the visitor to define what product by i does in the complex plane this would be quite cool.
A room with two walls closing on the visitor like in that famous star wars scene. But without the intergalactic trash. With ungraded calc exams instead.
A performance-theatre show. An artificial beach with Newton (with a wig) looking at a shell and at a pebble, figuring out which one is smoother. We hear a loud applouse and two giants carry him out of the building. The two giants come back to find a seminaked Archimedes computing an integral on the sand, they slaughter him.
This one will save mathematics from the financial crisis: A chair in which the visitor sits and his brain activity is monitored as he does some easy mathematics. Whenever his brain activity seams to resemble math thinking he is injected endorphins.
sorry, this got a bit out of control....
Im trying to think of something good for some basics about Galois theory or Covering Space theory, but this is harder....
Oh and in the store you get to draw your favorite planar graph and you leave with a clay model of the corresponding circle packing.