Note that we can use $g,h$ to code any real of $V$ in a recursive fashion (recursive in the Cohen reals added by $g$ and $h$). In particular we can use them to code $0^{\#}$ -- a real that cannot be added to $L$ via set forcing. This shows that $0^{\#}$ does in factindeed imply the failure of upward directedness of the set generic universe over $L$.
On the other hand, if $\mathrm{HOD}^V = L$, any set is generic over $L$ (via the Vopenka algebra) and hence the set generic universe over $L$ is upward directed.