Existence of triply orthogonal coordinates Definition
Consider a domain $\Omega \subseteq \mathbb{R}^3$ and three differentiable maps $f_i:\Omega \rightarrow \mathbb{R}$, $i = 1, 2, 3$.  If at every point $x \in \Omega$, $\nabla f_i(x) \cdot \nabla f_j(x) = 0$ whenever $i \ne j$, then the $f_i$ are triply orthogonal coordinates on $\Omega$.
Context
Dupin's theorem states that orthogonal coordinate surfaces (i.e., level sets of orthogonal coordinate functions) intersect along lines of principal curvature.  A classic example is Monge's ellipsoid -- see Jorge Sotomayor, "Historical Comments on Monge's Ellipsoid and the Configuration of Lines of Curvature on Surfaces Immersed in $\mathbb{R}^3$."
Question
Suppose we start with a compact, connected, orientable smooth surface $\mathcal{M}$ embedded in $\mathbb{R}^3$, and let $\Omega$ be a band of uniform thickness around $\mathcal{M}$ (i.e., a union of $\epsilon$-balls centered at each point on $\mathcal{M}$) small enough so that at each point $x \in \Omega$ there is a unique closest point on $\mathcal{M}$.  Further, suppose $f_1:\Omega \rightarrow \mathbb{R}$ gives the signed distance to $\mathcal{M}$, so that $f^{-1}(0) = \mathcal{M}$.
Can we always find (nontrivial) functions $f_2, f_3$ such that $(f_1,f_2,f_3)$ are triply orthogonal coordinates?
Or less formally, can we always find orthogonal coordinates "around" a given surface?
 A: It seems that this problem is studied under the heading of Lamé families, i.e., those families of surfaces belonging to an orthogonal triple.  Cayley was the first to give a general characterization of Lamé families in terms of a system of 3rd-order differential equations -- a simplified form can be found on p. 466 of A. R. Forsyth, "Lectures on the Differential Geometry of Curves and Surfaces".  In his paper, "On Lamé Families of Surfaces"  (1926), C.E. Weatherburn gives an even simpler characterization via the following condition on the unit normal field $n$:
$$ \mathrm{div}(\frac{\partial}{\partial n} \mathrm{curl}\ n) = 0.$$
In the paper, the author applies this condition as if it were sufficient to characterize a Lamé family (see, e.g., the paragraph on p. 303 following the statement of the theorem), but from the proof it seems like only a necessary condition.  In any case, in the book, "Differential Geometry of Three Dimensions" the same author states,

"A necessary and sufficient condition that a family of surfaces may form part of a triply orthogonal system is that, on each surface, the vector $\bar{\nabla}\psi$ be the gradient of some scalar function." (p. 101)

Here $\psi = ||n||$, and the operator $\bar{\nabla}$ is given by $\bar{\nabla}\phi = -\nabla \phi \cdot \nabla n$ where $\nabla$ is the surface gradient operator.  In particular (he notes), any parallel family of surfaces is a Lamé family.  So the answer to the original question is positive (since the distance function $f_1$ defines a parallel family).
A: I give an explicit counterexample below, so I have modified my answer: No, these sorts of orthogonal coordinates on 'shells' of parallel compact surfaces don't always (or even usually) exist.   The reason has to do with whether there are nonconstant functions on the given surface $\mathcal{M}$ that are constant on one of the families of principal curves. 
Such functions don't always exist: Suppose you take an oriented surface $\mathcal{M}\subset\mathbb{E}^3$, maybe, say, an embedded torus such that it has no umbilics and such that at least one family of its principal curves (i.e., the lines of curvature) wind densely around the torus.  It then turns out that, for $|t|<\epsilon$, the oriented-distance $t$ parallel surface $\mathcal{M}_t$ (where $\mathcal{M} = \mathcal{M}_0$) also has these properties.  An example is constructed below.  [Note that the standard ellipsoid is very different:  The lines of curvature on an ellipsoid are all closed.  However, this is very unusual behavior.  Most differential geometry books give the ellipsoid example because it's pretty and computable, and this tends to make students think that lines of curvature are 'usually' closed, but they are not.]
I claim that, for a surface as above, the desired functions $f_2$ and $f_3$ will not exist on the $\epsilon$-shell $\mathcal{S}$ around $\mathcal{M}$.  The reason is that each noncritical level set of $f_2=c$ (for example) will be closed in $\mathcal{S}$ and will have to intersect each $\mathcal{M}_t$ in a line of curvature, which, by supposition, is a dense curve in $\mathcal{M}_t$.  Thus, this (closed) level set will have to include $\mathcal{M}_t$.  Hence, this level set must be a disjoint union of the parallel surfaces $\mathcal{M}_t$.  It follows that $\nabla f_1$ and $\nabla f_2$ must be parallel along any noncritical level set of $f_2$.  However, this is absurd, because $\nabla f_1$ and $\nabla f_2$ must be perpendicular wherever $\nabla f_2$ is nonvanishing and $\nabla f_2$ does not vanish along a noncritical level set (by definition).  It follows, then that $f_2$ cannot have any noncritical level sets, i.e., $\nabla f_2$ must vanish identically.  Since you don't consider this to be a nontrivial solution, there are no nontrivial solutions. 
Added comment (an explicit example):  It turns out that there's an easy explicit example of a torus for which there is no nontrivial solution.   Let $C\subset\mathbb{E}^3$ be a closed, embedded, regular space curve, parametrized by arclength $ds$.  Thus, associated to the inclusion mapping $X:C\to\mathbb{E}^3$, there will be its Frenet apparatus, i.e., an orthonormal frame field $T,N,B:C\to S^2$ and functions $\kappa,\tau:C\to\mathbb{R}$ with $\kappa >0$ such that 
$$
dX = T\ ds,\quad dT = \kappa N\ ds,\quad dN = (\tau B -\kappa T)\ ds,\quad
dB = -\tau N\ ds.
$$
It is easy to see that there are such curves with $\int_C \tau\ ds$ being an irrational multiple of $\pi$, so let's assume this.  Now, for sufficiently small $t>0$, consider the tube of radius $t$ about $C$, parametrized by $Y_t:C\times S^1\to\mathbb{E}^3$ where 
$$
Y_t = X + t\cos\theta\ N + t\sin\theta\ B.
$$
(We require only that $t$ be so small that $Y_t$ be a smooth embedding of $C\times S^1$. In particular, we want $t$ to be less than $1/\kappa_{max}$.)  This is a parallel family of surfaces.
It is easy to compute that the principal curves on $Y_t$ are the leaves of the two foliations $ds=0$ and $d\theta + \tau(s)\ ds = 0$.  Because I required that $\int_C\tau\ ds$ be an irrational multiple of $\pi$, it follows that each of the leaves of $d\theta + \tau(s)\ ds = 0$ is dense in $C\times S^1$.  Consequently, the above argument applies to show that, even though you can take $f_1$ to be $t$ for this family, and, say $f_3$ to be some function of $s$, any function $f_2$ on a shell constructed by taking $t$ in some small interval in $(0,1/\kappa_{max})$ that satisfies $d f_2 \wedge(d\theta + \tau(s)\ ds) =0$ will have to be constant on the level sets of $f_1$.
One further remark: It turns out that a necessary and sufficient condition for the existence of a non-trival solution $(f_2,f_3)$ for the given $f_1$ is simply this:  A nontrivial (in the OP's sense) solution exists if and only if there exist functions $g_2$ and $g_3$ on $\mathcal{M}$ with the properties that $dg_2\wedge dg_3$ is nonvanishing on a dense open set in $\mathcal{M}$ and that each $g_i$ is constant on a family of principal curves on $\mathcal{M}$.  Sufficiency follows since one can simply take $f_i = \pi^*(g_i)$ for $i=2$ or $3$ where $\pi:\mathcal{S}\to\mathcal{M}$ is the retraction of the shell $\mathcal{S}$ to $\mathcal{M}$ long the normal lines from $\mathcal{M}$.  Necessity follows since, if $f_2$ and $f_3$ exist, then one can simply define $g_i$ to be the restriction of $f_i$ to $\mathcal{M}$.  Thus, one can see that the behavior of the principal curves on $\mathcal{M}$ completely determines whether or not there are solutions.
A: Unless I misunderstand your question, the functions $f_2, f_3$ yield sections of the tangent bundle of $\mathcal{M}$ which will span the tangent space at any point i.e. a necessary condition is that your surface is parallelizable (i.e. it has a trivial tangent bundle).
A: (Sorry if you saw my original answer but it was completely wrong)
I don't know the answer to this question, because I believe that it is equivalent to an initial value problem for a system of PDE's, where the initial data is posed on a characteristic surface. My guess is that the answer is in general no, but it is just a guess.
However, if you pose the initial data differently, then there is the following theorem: If instead of specifying the function $f$, you specify an 3-d orthonormal frame along the surface so that none of the vectors in the frame are ever tangent to the surface, then there exist triply orthogonal co-ordinates (i.e., co-ordinates such that the metric tensor is diagonal) in a neighborhood of the surface such that their gradients point in the same directions as the given orthonormal frame. This is proved in: Existence of elastic deformations with prescribed principal strains and triply orthogonal systems. Duke Math. J. 51 (1984), no. 2, 243--260. 
