You are right.
In the case of an $R$-submodule $D$ of an $R$-coalgebra $C$, the correct definition for $D$ being a subcoalgebra of $C$ is your definition (2) and not the one posted in your notes. This is standard in the contemporary literature: see for example p.11, sect. 2.7 of 1.
The definition mentioned in your notes is valid (as you have already mentioned in the OP) for vector spaces and special cases for $R$-coalgebras (when $D$ is flat or pure as an $R$-submodule for example).
The problem is indeed the one you mention: In general, the restriction of the comultiplication of $C$ to $D$, that is $\Delta\vert _D: D \to C \otimes C$, does not necessarily "lift" to a map $\Delta\vert _D: D \to D \otimes D$. This can happen, even in cases in which $\Delta(D)$ is contained inside the image of the canonical map $D\otimes_R D\to C\otimes_R C$.
A relevant example (attributed to Warren Nichols) is the following: Consider the $\mathbb{Z}$-module $C=\mathbb{Z}/8\mathbb{Z}\oplus \mathbb{Z}/2\mathbb{Z}$ and denote $x=(1,0)$, and $z=(0,1)$. A comultiplication $\Delta:C\to C\otimes_\mathbb{Z} C$ is defined by $\Delta(x)=0$ and $\Delta(z)=4x\otimes x$. So $\Delta(z)$ has order $2$ in $C\otimes_\mathbb{Z} C$. Then, let $y=2x$ and take the $\mathbb{Z}$-submodule $D=\mathbb{Z}y+\mathbb{Z}z\subset C$. Now, $\Delta(z)=y\otimes y$ and $\Delta(D)$ is contained in the image of $D\otimes_\mathbb{Z} D\to C\otimes_\mathbb{Z} C$ but the restriction $\Delta|_{D}$ does not lift (or: does not factor) to a comultiplication $\Delta:D\to D\otimes_\mathbb{Z} D$, since we can see that any preimage of $y\otimes y$ in $D\otimes_\mathbb{Z} D$ has order $4$.
(This example was first suggested by Nichols in 2, p.56 and is also cited as an exercise in 1).
Edit: Another interesting phenomenon, related to the definition, of an $R$-subcoalgebra (definition (2) of the OP), is the fact that following this, the $R$-subcoalgebra $D$ of the $R$-coalgebra $C$ is not uniquely determined: In other words, the same $R$-submodule may correspond to non-isomorphic subcoalgebras. This has already been mentioned and sufficiently explained in Adrien's answer. I thought it might be interesting to include an example of this "phenomenon" (which is again suggested in 2 and cited as an exercise in 1):
Consider the $\mathbb{Z}$-module $C=\mathbb{Z}\oplus\mathbb{Z}/4\mathbb{Z}$ and make it a coalgebra by setting $g=(1,0)$ to be a grouplike and $x=(0,1)$ to be $g$-primitive. Then, take the submodule $D\subset C$ which is spanned by $g$ and $2x$. So, $D\cong\mathbb{Z}\oplus\mathbb{Z}/2\mathbb{Z}\cong\mathbb{Z}\oplus 2\mathbb{Z}/4\mathbb{Z}$. There are (at least) two different ways to make $D$ into a coalgebra: Either consider $g$ to be a grouplike and $2x$ to be $g$-primitive (this corresponds to the restriction of the original map of $C$ in $D$), or take again $g$ to be grouplike and define
$$
\Delta(2x)=g\otimes(2x)+(2x)\otimes(2x)+(2x)\otimes g \ \ \ and \ \ \ \epsilon(2x)=0
$$
Both options make $D$ into a coalgebra and in both cases one can easily show that the inclusion $D\hookrightarrow C$ is a coalgebra morphism. So, $D$ becomes a $\mathbb{Z}$-subcoalgebra of $C$ in (at least) two different ways. Furthermore, these are not isomorphic to each other, since the second structure has an additional grouplike element; this is $g+(2x)$.
References:
- Corings and comodules, T. Brzezinski, R. Wisbauer,
- Nichols, W.D., Sweedler, M., Hopf algebras and combinatorics, AMS, in: "Umbral Calculus and Hopf algebras", Contemp. Math. 6, 49–84 (1982)