I'll assume $X$ and $Y$ are to be independent, and instead of "given $p$" you mean you want a parametric family of probability density functions (apparently on $(0,\infty)$?) for all $p \in (0,1)$.
This is very easy. For each $p \in (0,1/2]$ you take an arbitrary pdf $f(\cdot, p)$, and
then for $p \in (1/2, 1)$ you just need some pdf $f(\cdot, p)$ so that $\int_0^\infty f(x,p) \int_x^\infty f(y,1-p)\ dy \ dx = p$. Since that integral is near $1$ if $f(\cdot,p)$ is
concentrated near $0$ and near $0$ if $f(\cdot, p)$ is concentrated on very large values, by the Intermediate Value Theorem any continuous one-parameter family that goes between these two extremes will have a member that makes the integral equal $p$.
For a specific easily computed example, note that if $X$ and $Y$ are independent exponential random variables with rates $r_1$ and $r_2$ respectively, $P(X < Y) = r_1/(r_1 + r_2)$.
So we could take $f(\cdot,p)$ to be the exponential density with rate $r(p)=p$, i.e.
$f(x,p) = p e^{-px}$ for $x > 0$, $0$ otherwise.