Upon the OP's suggestion, here is an expansion of my comments that shows $C_p < 2$.

Let $C_p$ be above. We look at two cases: (i) $1 < p < 2$, and (ii) $p > 2$.

Case (i): From [this note here][1] we know that
\begin{equation*}
\|x+y\|_p^2 \le 2(\|x\|_p^2 + \|y\|_p^2) + (1-p)\|x-y\|_p^2.
\end{equation*}
Using the hypothesis, $\|x\|_p=\|y\|_p=\|x-y\|_p=1$, we obtain
\begin{equation*}
 \|x+y\|_p^2 \le 5 - p\quad\implies C_p < 2.
\end{equation*}

Case (ii): [From Hanner's inequalities][1] since $p > 2$ we know that 
\begin{equation*}
\|x+y\|_p^p + \|x-y\|_p^p \le (\|x\|_p+\|y\|_p)^p + | \|x\|_p - \|y\|_p |^p.
\end{equation*}
Using the hypothesis, we obtain
\begin{equation*}
 \|x+y\|_p \le (2^p - 1)^{1/p},\quad \implies C_p < 2.
\end{equation*}

The only case we skipped here is $p=2$, but in that case $C_2=\sqrt{3}$ is already known to the OP.
  [1]: https://en.wikipedia.org/wiki/Hanner's_inequalities