Upon the OP's suggestion, here is an expansion of my comments that shows $C_p < 2$.
Let $C_p$ be above. We look at two cases: (i) $1 < p < 2$, and (ii) $p > 2$.
Case (i): From [this note here][1] we know that \begin{equation*} \|x+y\|_p^2 \le 2(\|x\|_p^2 + \|y\|_p^2) + (1-p)\|x-y\|_p^2. \end{equation*} Using the hypothesis, $\|x\|_p=\|y\|_p=\|x-y\|_p=1$, we obtain \begin{equation*} \|x+y\|_p^2 \le 5 - p\quad\implies C_p < 2. \end{equation*}
Case (ii): [From Hanner's inequalities][1] since $p > 2$ we know that \begin{equation*} \|x+y\|_p^p + \|x-y\|_p^p \le (\|x\|_p+\|y\|_p)^p + | \|x\|_p - \|y\|_p |^p. \end{equation*} Using the hypothesis, we obtain \begin{equation*} \|x+y\|_p \le (2^p - 1)^{1/p},\quad \implies C_p < 2. \end{equation*}
The only case we skipped here is $p=2$, but in that case $C_2=\sqrt{3}$ is already known to the OP. [1]: https://en.wikipedia.org/wiki/Hanner's_inequalities