Before I come to the core of your question concerning the subgroup $\mathrm{Diff}(M,S)$ let us first clarify the terms Lie group and subgroup:
In general there are many different concepts, especially since you are dealing with a Lie group beyond the Banach setting. Since you are only interested in Fréchet Lie groups at least there is no (real) choice of infinite-dimensional calculus as the two most popular and widely used offerings [Convenient calculus (developed thoroughly in 1, see also chapter 43 for a discussion of the diffeomorphism group in this setting!) and Bastiani calculus (see e.g. the survey 2 for references and a discussion of Lie groups in this setting)] coincide.
Now we can define manifolds and Lie groups (=group with manifold structure s.t. group operations become smooth) in the usual way and one may ask what a Lie subgroup is.
I will follow here Neeb's survey 2 (still one of the best places to learn about infinite-dimensional Lie groups even beyond the Fréchet setting!) and the following numbers reference his survey. Mainly two concepts are discussed in 2:
- Lie subgroups, meaning a submanifold $H$ of a Lie group $G$ which is a subgroup, such that the submanifold structure turns it into a Lie group (cf. Remark II.2.5 (a), submanifold means here that there is a closed subspace $F$ of the model space of $G$ together with a (submanifold) atlas mapping $H$ into $F$)
- initial Lie subgroups, meaning an injective morphism $\iota \colon H \rightarrow G$ of Lie groups such that induced morphism between the Lie algebras is injective, and a map $f\colon K \rightarrow H$ from any $C^k$-manifold $K$ is $C^k$ if and only if the corresponding map $\iota \circ f$ is $C^k$ (see Section II.6).
I just mention the initial Lie subgroup property only because it is a quite weak property. In particular 1. implies 2. and while in many places in the literature one can find definitions of Lie subgroups as in 1. (but for a variety of submanifold concepts!!! See e.g. the Bourbaki book on Lie groups (which works in a Banach setting)) these concepts almost always imply that the Lie subgroup is an initial Lie subgroup. Anyway 2 contains a more in depth discussion of this.
Having now briefly discussed Lie subgroups, lets get back to your initial question about $\mathrm{Diff}(M,S)$. First the bad news:
Indeed the question of how and when this subgroup becomes a Lie subgroup does not seem to be well studied and seems to be difficult. The reason for this is the way one constructs the manifold structure on $C^\infty (M,M)$ (of which $\mathrm{Diff} (M)$ is an open submanifold, see e.g. 3): The charts of the manifold of mappings $C^\infty (M,M)$ are not induced by charts of the target manifold $M$ but one needs to construct them using an auxiliary tool called a local addition (this has been explicitly discussed elsewhere, probably also several times on MO, so I am not going into details here). Bottom line is however that one can not simply ''lift the submanifold charts of $S$'' to obtain submanifold charts of $\mathrm{Diff}(M,S)$.
So is all hope lost to get something sensible out of the construction? Fortunately, there is a construction which at least point in the right direction: In their seminal paper ''Groups of Diffeomorphisms and the Motion of an Incompressible Fluid'' 4 Ebin and Marsden deal with such questions in Section 6. For example Theorem 6.1 states:
Let $M$ be a compact manifold (without boundary) and $N\subseteq M$ a closed submanifold without boundary. Let $$D_N := \{\phi \in \mathrm{Diff} (M) | \phi (N) \subseteq N\}, \qquad D_{N,p} := \{\phi \in \mathrm{Diff} (M) | \phi (x) = x, \forall x\in N\}$$
Then $D_N$ and $D_{N,p}$ are ILH Lie subgroups of $\mathrm{Diff} (M)$. (Further their Lie algebras are identified)
Now an ILH Lie group is a certain inductive limit of Hilbert Lie groups (concept is due to Omori). However, on p.117 (directly after the statement of the Theorem), Ebin and Marsden state that their proof actually shows that these groups are actually submanifolds of $\mathrm{Diff} (M)$ (whence they satisfy definition 1. from above). Moreover, in Proposition 6.8 of 4 they prove for a compact manifold $M$ with boundary that the unit component of $\mathrm{Diff} (M)$ embedds as a submanifold (whence is a Lie subgroup!) of the group $\mathrm{Diff}(DM)$, where $DM$ is the double of $M$. (For this argument one needs an extension theorem for vector fields such as the Whitney extension theorem 5 or the Calderón extension theorem 4 for Sobolev class morphisms.)
The main point of these constructions (extracted again from 4) is that (in the above cases) one can construct a Riemannian metric on the finite-dimensional manifold $M$ such that the submanifold $N$ becomes a totally geodesic submanifold. If one can do this, then the Riemannian exponential map of the metric yields a local addition (remember the above remarks where I omitted details) which then restricts to a submanifold chart for $\mathrm{Diff}(M,S)$.
So my best guess is that similar arguments will work if you have a Riemannian metric which turns the submanifold $S$ into a totally geodesic submanifold (Observe that the manifold structure on $\mathrm{Diff} (M)$ does not depend on the choice of Riemannian metric, this is purely auxiliary!). However, this is indeed the only reference I am aware of, which discusses groups as in your question.
Short edit: The point of having a totally geodesic submanifold is that the local addition (given by the Riemannian exponential map) on $M$ restricts to a local addition on the submanifold $S$. So in case you are working with some other choice of local addition, the property you want to require to obtain a Lie subgroup is that the local addition of $M$ restricts to a local addition on $S$.
