Separable coordinate systems for the Laplace and Helmholtz equations? According to Mathworld, in three dimensions there are 13 coordinate systems in which Laplace's equation is separable, and 11 for the Helmholtz equation. I've read the relevant chapters of the book by Morse & Feshbach. Apart from a recent paper by Phil Lucht, almost nothing has been written about this since the sixties. The situation is actually more complicated and the numbers fairly subjective: 
For Helmholtz there is a whole class of separable coordinate systems based on quadrics. For Laplace, all of these are separable, but there are also coordinate systems based on cyclides. 


*

*Do any of the works mentioned above constitute a proof that these are the only separable coordinate systems in three dimensions? My impression is that they don't give a rigorous proof. Is there another book or paper that proves this? 

*Has anyone enumerated all the separable coordinate systems in four and higher dimensions? 

*Given that this work hasn't already been done, would it be worthwhile for me to do it? I have done a bit of the work already, and I'm interested in doing more. But I'd like to know if this would be a publishable result. The fact that (apparently) hardly any work has been done in this area in the last sixty years suggests to me that maybe people just aren't interested. 

 A: Separation of variables is a classical subject with much written on it, even in recent years, though in much more specialized literature. I'm not personally very familiar with proofs of exhaustive classifcations of separable coordinate systems, but a discussion of that can be found in several places. One of them, I think is in this book:

Miller, W. Symmetry and Separation of Variables, vol. 4 of Encyclopedia of Mathematics and its Applications (Addison-Wesley, 1977)

In my understanding the starting point for exhaustive classifications is this seminal work of Eisenhart, and of course its generalizations, on the relation between separable coordinate systems for the Laplace equation and the existence of Killing tensors:

Eisenhart, L. P. Separable systems of Stäckel, Annals of Mathematics 35, 284-305 (1934)


Update: Here are some more references on the classification of separable coordinate systems for several classical types of second order equations, including some work on higher dimensional spaces. It seems that work needed to classify the separable coordinate systems in higher dimensions is very time consuming, so using a computer is recommended. I do not claim to be very familiar with the content of these references, so you may need to glance through them to see if they do or do not contain the results that are of interest to you.

Boyer, C. P., Kalnins, E. G. & Miller, W. Symmetry and separation of variables for the Helmholtz and Laplace equations, Nagoya Mathematical Journal 60, 35-80 (1976)
Kalnins, E. G., Miller, W. & Williams, G. C. Separability of Wave Equations, chap. 3, 33-52 (Springer Netherlands, Dordrecht, 1999)
Reid, G. J. R-separation for heat and schrödinger equations I, SIAM Journal on Mathematical Analysis 17, 646-687 (1986) Note: Treats higher dimensions.
Kalnins, E. G., Miller, W. & Reid, G. J. Separation of variables for complex riemannian spaces of constant curvature. I. orthogonal separable coordinates for S$_{n\mathbb{C}}$ and E$_{n\mathbb{C}}$, Proceedings of the Royal Society of London A: Mathematical, Physical and Engineering Sciences 394, 183-206 (1984) Note: Treats higher dimensions.

There appears to be much less written on equations that don't fall into one of the classic second order equations, but here's at least one reference that I am aware of that also discusses higher order equations:

Hainzl, J. On a general concept for separation of variables. SIAM Journal on Mathematical Analysis 13, 208-225 (1982)

The question of whether the work that you have in mind has already been done and, if not, whether it is worth doing seems difficult to answer for a non-expert. From what I know, separation of variables (or some particular aspects of it) is still interesting to people working in so-called integrable systems. The separability of specific equations is also of interest in physics (e.g., wave equations on black hole and cosmological spacetimes). If you can locate an active expert, for example by following up on some of the above literature, you may get a more specific answer.
A: (CW because I don't say anything "mathematical" [i.e. answers questions 1 and 2]; so the below is a subjective and biased opinion on question 3.)
Would it be publishable of you solve the problem?
Given the state of the publishing industry nowadays, hardly anything is "not publishable". So I assume you mean publication in a suitably serious journal. That is then tied to the next question...
Would it be worthwhile?
First you should ask yourself why was this question asked in the first place. So what does a separation of variables give you? One big advantage is in the construction of explicit solutions. In 1D the problem of computing the spectrum of perturbations of the Laplacian is immensely more tractable than in higher dimensions in general, and the associated generalized eigenfunctions can be obtained by solving ODEs. This makes it much easier to actually go in and by hand do the Fourier decomposition. 
As an example, on an arbitrary compact Riemannian three manifold we know we can decompose any function $f$ into eigenfunctions of the Laplacian. But if you give me a three manifold and a function and ask me to compute the Fourier coefficients and the eigenfunctions, I can just through my hand up in the air and say: "no can do, unless you want numerical results". If you work on the torus, however, things can be done explicitly because of the nice separation of variables. 
So separation of variables for the Laplace and Helmholtz equations give us a method of explicitly solving the Poisson equation and the Cauchy problem for the wave equation using this method .... on Euclidean spaces, however, these can also be done by convolving with the explicit integral kernel. Before computers are widely used, the former has the appearance of giving a larger class of explicit solutions. 
In terms of PDE theory, we have moved quite far from "separation of variables for Laplace/Helmholtz". Part of it is because the study of linear equations with constant coefficients are by now considered largely uninteresting: modern studies focus much more on PDEs with variable coefficients, nonlinearities, or both. (I should note that the importance of considering variable coefficients, in terms of equations with potentials, was expected even in the 40s.) For nonlinear equations, in particular, the classical separation of variable ansatz becomes of rather limited use. 
This is not to say that people in pure PDEs don't care: just that you have to realign your paper and your goals to the current interest. So in terms of variable coefficients it may be more worthwhile to study separation of variables on Riemannian or pseudo-Riemannian manifolds, with respect to their Laplace-Beltrami operators, rather than restricting yourself to higher dimensional Euclidean spaces. Or you can dive deeper in some of the more modern developments that are, at some point, connected to separation of variables techniques. For example you can study symmetry properties of differential equations (see Olver's book for a starting point). Or you can study the problems from the point of view of "generalised-" or "functional-" separation of variables.
Another possibility is of course to consider the problem less from the pure point of view but more in terms of its (possibly computational or engineering) applications. For that I am the wrong person to ask. But based on the publications of various handbooks on linear PDEs, there does appear to still be substantial interest in methods of solving linear PDEs from the engineering community. 
A: Just to clarify things, there are 17 conformally inequivalent coordinate systems which separate the Laplace equation in three-dimensions. This information can be found for instance in Section 3.6 the previously mentioned monograph: Miller, W. Symmetry and Separation of Variables, vol. 4 of Encyclopedia of Mathematics and its Applications (Addison-Wesley, 1977), in the chapter on Separation of Variables. As you said, there are 11 quadric (second order) coordinate systems which separate the 3-variable Helmholtz equation, but for the Laplace equation, there is an additional 6 conformally inequivalent cyclidic (4th order) coordinate systems: toroidal, flat-ring cyclide, flat-disk cyclide, bi-cyclide, and there are also two asymmetric cyclidic coordinate systems which generalize the quadric ellipsoidal and paraboloidal coordinates.
