Is symplectic reduction interesting from a physical point of view? Do you think that symplectic reduction (Marsden Weinstein reduction) is interesting from a physical point of view? If so, why? Does it give you some new physical insights?
There are some possible answers I often heard of, but I don't really understand it. Perhaps you could comment these points. Explain and illustrate why they are good reasons or if not, explain why they are nonsense:
1) Symplectic reduction is interesting because "it simplifies" the system under consideration because you exploit symmetries to eliminate some redundant degrees of freedom. I do not really understand what's the point here because in general reduction leads to a more complicated geometry. (Or even to singular spaces if you consider more general reduction settings).
2) It's interesting because it is a toy model for gauge theories. 
3) It's interesting because if you want to "quantize" a system, from a conceptual point of view, one should start from the reduced system, from the "real" phase space. I don't see why one should do this for nonrelativistic quantum systems. Even for gauge theories I don't get the point, because the usual procedure is quantize the unreduced system (via gauge fixing), isn't it?
If there are points which make symplectic reduction interesting from a physical point of view, are there physical reasons why one should study reduction by stages?
Added; after reading José Figueroa-O'Farrill's answer I had some thoughts I should add:
I am still by far not an expert in gauge theories. But I think, that in gauge theories, one typically has redundant variables, which have no or at least not a direct physical interpretation. So I would agree that the "physical" dynamics takes place at the quotient in the case that the gauge theory itself has a physical meaning (in particular experimental evidence). Concerning quantization, however, if I am not mistaken, the only known quantum gauge theory which corresponding classical gauge theory which experimental support is quantum electrodynamics. For the other physical relevant quantum gauge theories, I think, the classical counterparts play just the role of auxiliary theories in some sense. In this case I would agree, that on the quantum side only the reduced space has physical meaning, but I think for the corresponding classical one, this seems to be a rather pointless question. So the question remains, why it is physically interesting to study on the classical side the reduced phase space in case of gauge theories. Moreover as José Figueroa-O'Farrill points out, the classical reduced space in most cases too complicated to quantize it directly, one would use some kind of extrinsic quantization as BRST instead. I don't know exactly how the situation for gravitation is. I think one can formulate ART as classical gauge theory. But makes it sense in this case to study the reduced classical phase space for quantization purposes? I guess not.
 A: 
Symplectic reduction is interesting
  because "it simplifies" the system
  under consideration because you
  exploit symmetries to eliminate some
  redundant degrees of freedom. I do not
  really understand what's the point
  here because in general reduction
  leads to a more complicated geometry.
  (Or even to singular spaces if you
  consider more general reduction
  settings).

Actually, the idea is to a find simpler dynamic "upstairs" in order to understand the complicated dynamic that occurs on the quotient space. 
Let me give you an example with the so-called Calogero-Moser system. 
Consider the cotangent space $T^*(\mathbb{C}^n_{reg})$ of the space $\mathbb{C}^n_{reg}$ consisting of $n$ pairwize dinstinct points in the complex plane. And try to study the dynamic associated to the Hamiltonian 
$$
H(p_1,\dots,p_n,q_1,\dots,q_n):=\sum_ip_i^2-\sum_{i\neq j}\frac{1}{q_i-q_j}
$$
Are there enough conserved quantities ? What are the integral curves ? etc...
Actually you can see that everything is invariant under the symmetric group $S_n$ so I will actually try to study the same dynamic on $T^*(\mathbb{C}^n_{reg})/S_n$. 
An a priori unrelated system
Consider the space of pairs $(X,Y)$ of $n\times n$ matrices. We actually have $M_n\times M_n=T^*(M_n)$ via the bilinear form $tr(XY)$. The Poisson bracket on coordinates is given by 
$$
\{x_{ij},x_{kl}\}=0=\{y_{ij},y_{kl}\}\quad\{y_{ij},x_{kl}\}=\delta_{il}\delta{jk}
$$
Where $x_{ij}$ and $y_{kl}$ are the obvious coordinates on $M_n\times M_n$. 
Consider the map to $\mathfrak{sl}_n$ defined by $\mu(X,Y)=[X,Y]$. It is a momentum map ( we again identify $\mathfrak{sl}_n$ with its dual as above), and take the reduction w.r.t the (co)adjoint orbit $\mathcal{O}$ of $diag(-1,\dots,-1,n-1)$. The reduced space $M_{red}$ is then the space of pairs $(X,Y)$ of matrices such that $[X,Y]-Id$ has rank $1$, modulo simultaneous conjugation. 
Now observe that the functions $H_i=tr(Y^i)$, $i=1,\dots,n$, form an integrable system on $M_{red}$. Namely, they are independant Poisson commuting and conjugation invariant functions on $M_n\times M_n$ - therefore they descend to $M_{red}$ which has precisely dimension $n$. 
The dynamic of $H_2$ is very easy. It is linear! Integral curves of $H_2$ are of the form 
$$
(X(t),Y(t))=(X_0+2Y_0t,Y_0). 
$$
How the hell is this related to what we had before?
The point is that we have an injective Poisson map from $T^*(\mathbb{C}^n_{reg})/S_n$ to $M_{red}$ that sends $(p_1,\dots,p_n,q_1,\dots,q_n)$ to the pair $(X,Y)$ with 
$X=diag(q_1,\dots, q_m)$, $Y_{ii}=p_i$ and $Y_{ij}=\frac{1}{q_i-q_j}$. Moreover, th eimage of this map coincide with the dense open subset consisting of conjugation classes of pairs $(X,Y)$ such that $X$ is diagonalizable with pairwize distinct eigenvalues. 
Now if you write $H_2$ in $(p,q)$ coordinates you find exactly the $H$ we started with. 
So, in the end we found a way to write a complicated system as the reduction of a very simple one. This helps us to understand well the complicates one (e.g. in this example it helped to prove integrability). 

The is not the only motivation for symplectic reduction. Studiing systems with constraints might be another one. But I found this example very enlighting about the potential useness of symplectic reduction. 

reference: I think that the people who did this are Kazhdan, Kostant and Sternberg. You can find a very nice presentation of it (as well as many other interesting things) in the book "Lectures on Calogero-Moser systems" by Pavel Etingof. 
A: Of course it is interesting, and the idea of factoring out symmetries goes back to Newton.
As for your point (1), yes it leads to more complicated geometry, and often some form of singular reduction is required (e.g. have a look at the blue book of Bates & Cushman for the case of integrable Hamiltonian systems with finitely many degrees of freedom), but from a dynamical point of view it makes much more sense. 
For instance suppose you would like to study numerically a classical mechanical system (integrable or non-integrable all the same): working in reduced coordinates allows to easily distinguish between different Periodic Orbits (i.e. to count them only once), while in non-reduced dynamics the Periodic Orbits come in continuous families. 
As for your comment, the topology of the reduced space is certainly a "new physical insight". Have a look at this recent paper by Yanguas, Palacian, Meyer & Dumas where they study periodic orbits of a non-integrable system as you ask, and where they discuss the issue of reduction and provide further references.
(Edited to correct an error; more links added in reply to comment.)
A: Here is a fancy example: Supersymmetry. Rigid N=1 supersymmetric theories in 4 dimension have a natural Kahler structure on the field space. The D-term is precisely a moment map. The moduli space of the theory is the symplectic quotient from this moment map.
A: Symplectic reduction arises naturally in constrained hamiltonian systems, e.g., gauge theories.  So it is not just a question of it being "interesting" as much as a fact of life.
The way to deal with coisotropic constraints -- those whose zero locus is a coisotropic submanifold -- is via symplectic reduction.  The real (read, physically meaningful) dynamics are taking place in the symplectic quotient, which is the standard quotient of the zero locus of the constraints by the (integrable) distribution defined by the hamiltonian vector fields associated to the constraints.
Now, as you point out, the symplectic quotient is usually much more complicated geometrically than the original symplectic manifold and this makes working there cumbersome.  For instance, quantising the symplectic quotient is usually difficult.  Luckily, one can go the other way: instead of performing the symplectic quotient and then quantising, one can first quantise the constrained system and then do a quantum version of the symplectic quotient.  One such procedure, which works in many gauge theories, is BRST quantisation.  This is a homological approach to the quantisation of constrained hamiltonian systems.  It has the virtue that it preserves the symmetries of the original system which "gauge fixing" typically destroys.
A: Inviscid fluid mechanics is one example of a physical system where symplectic reduction actually tells you a great deal, which would be very hard to obtain by other means.
The unreduced configuration space for a (inviscid, incompressible) fluid in a 3-dimensional container $M$ is the group $SDiff(M)$ of volume-preserving diffeomorphisms.  This group acts on itself from the right and this action leaves the fluid kinetic energy invariant.  Now, there are various kinds of reduction, and all of them give some insight into fluid dynamics:


*

*Working directly with the unreduced configuration space allows you to do fluid dynamics in Lagrangian variables, that is, you track every particle through its motion and record where it goes as time progresses;

*By factoring out the diffeomorphism group action, you end up from $T^\ast Q$ in the dual of the Lie algebra of $SDiff(M)$ and this will let you do fluid dynamics in the Eulerian picture.  Here you are fixed in space, and instead of tracking individual fluid particles you record what happens at a fixed position in space.  This is an instance of Poisson reduction, and will show you that the canonical Poisson structure on the dual of the Lie algebra of $SDiff(M)$ is the right one if you want to show that Euler's equations are Hamiltonian.  Not only does this process explicitly give you the Poisson structure, it also tells you that (e.g.) it satisfies Jacobi's identity (since it is obtained through reduction from the canonical Poisson structure on $T^\ast Q$), a fact which would otherwise be hard to obtain directly.

*Now fix an element $\mu$ in the dual of the Lie algebra of $SDiff(M)$ and do symplectic reduction at $\mu$.  V.I. Arnold has shown that the elements of the dual of the Lie algebra of $SDiff(M)$ can be interpreted as vorticity distributions of the fluid, so performing symplectic reduction is tantamount to fixing the vorticity distribution of the fluid, and only considering fluids with that amount of vorticity.  One important example shows up when you take $\mu$ to have support along a closed curve (knot or link).  In that case, the symplectic reduced space is nothing but the space of all knots/links that are diffeomorphic to the original curve, equipped with the natural symplectic form!  This is quite a simplification, and for instance tells you that the dynamics of vortex rings (a fundamental part of fluid dynamics) is Hamiltonian and has an incredibly nice geometric interpretation.   Moreover, by varying $\mu$ you get various other fluid-dynamical systems, such as the Kelvin, Lamb, Kirchoff-Lin, etc. equations, all in one fell swoop!
It's also important emphasize that symplectic reduction doesn't just give you "old" results in a new formulation, but also clears up much of the confusion that you would get when doing straightforward calculations.  For instance, people have been known to come up with various sequences of "conserved quantities" for the Euler equations, attempting to establish complete integrability this way.  However, having a clear picture of the geometry and how the coadjoint orbits sit inside the dual Lie algebra will quickly show you that these quantities are merely Casimirs and could hence never be used for integrability.
A: Suppose you have a kinematics that is parameterized by a; whose symmetries are given by the Lie Algebra [X,Y] = aZ, [Y,Z] = X, [Z,X] = Y. For instance, a could be a coefficient that distinguishes a "classical" from "non-classical" formulation; or a parameter that distinguishes a "bound" (a < 0) from a "free" (a > 0) system. What are the representations of this system? And in what sense is "representation" meant?
With symplectic reduction one can not only answer the question within a unified framework in a way that cuts across paradigm boundaries (both quantum and classical); but across the different cases of the parameter (a).
In contrast with Hilbert space representation theory, where the different cases of (a) have to be treated separately and one has to resort to the awkward "group contraction" mechanism to handle the different cases, it is possible to devise the Poisson-Lie manifold for each case of (a) and to then continuously connect them all into a single Poisson manifold, by simply treating (a) as an extra coordinate.
The symplectic reduction is then carried out on the unified manifold. Each of the symplectic leaves for values of (a > 0) corresponds to irreducible representations. The method however generally breaks down when passing over to limit values of the contraction parameter (here: where a = 0), where the boundary case may not only fail the Peter Weyl Theorem, but may not even have a tractible Hilbert space representation problem at all. (Some groups, like the Galilei group have what's known as a wild representation type, whose representation problems are generally unsolvable.)
In contrast, passing over from a > 0 to a = 0 and even a < 0 causes no problem with symplectic reduction. For this example, the reduction yields the cases:
{(0,0,0,a)}, for all values of a
((0,0,Z,0)}, for all non-zero values of Z
{(X,Y,Z,a): XX + YY + aZZ = RR }, for all values of a and R > 0
and you can see immediately which subsets are connected to which, as (a) varies (i.e. how the symplectic leaves morph as (a) crosses from a > 0 to a = 0 to a < 0); and which ones are connected to which for each given value of (a).
This construction transcends the linear representation theory of Hilbert spaces in the sense that the Poisson manifold is not the Poisson-Lie manifold of any Lie group or its Lie algebra, but is a manifold which encapsulates an entire family of such Lie groups and Lie algebras in a single unified setting.
