The following mostly answers your question, but not completely:

1. $\newcommand{\C}{\mathcal{C}}\epsilon_X : \C(-,F(X)) \to [\C'](-,X)$ is injective because $\epsilon_X$ is mono, and is surjective by the definition of $[\C']$ together with the universal property of $\epsilon_X$ in the adjunction.

2. If $\C'' \subseteq \C' \subseteq \C$ are successively right rejective, with adjoints $F'$, $F$ to their inclusions $i'$, $i$, then the inclusion $i'' : \C'' \subseteq \C$ has a right adjoint given by $F' \cdot F$, as Julian Kuelshammer says in comments.

However, it is not clear to me why in (2.) the counit $\epsilon''$ will be pointwise mono. We have $\epsilon''_X = \epsilon_X \cdot \epsilon'_{FX} : F'FX \to X$, and $\epsilon_X$ is mono, but we only know that $\epsilon'_{FX}$ is mono in $\C'$, not necessarily in $\C$. (Subcategory inclusions don’t necessarily preserve monos.)

The following mostly answers your question, but not completely:

1. $\newcommand{\C}{\mathcal{C}}\epsilon_X : \C(-,F(X)) \to [\C'](-,X)$ is injective because $\epsilon_X$ is mono, and is surjective by the definition of $[\C']$ together with the universal property of $\epsilon_X$ in the adjunction.

2. If $\C'' \subseteq \C' \subseteq \C$ are successively right rejective, with adjoints $F'$, $F$ to their inclusions $i'$, $i$, then the inclusion $i'' : \C'' \subseteq \C$ has a right adjoint given by $F' \cdot F$, as Julian Kuelshammer says in comments.

However, it is not clear to me why in (2.) the counit $\epsilon''$ will be pointwise mono. We have $\epsilon''_X = \epsilon_X \cdot \epsilon'_{FX} : F'FX \to X$, and $\epsilon_X$ is mono, but we only know that $\epsilon'_{FX}$ is mono in $\C'$, not necessarily in $\C$. (Subcategory inclusions don’t necessarily preserve monos.)

Possible conditions that would give this are:

• any functor that preserves finite limits (or even just pullbacks) preserves monos

• if the inclusion has a left adjoint as well as a right adjoint, then the inclusion must preserve all limits