The answer to the second question is yes for simplicial polyhedra, i.e., those with triangular faces. Indeed one may use Alexandrov's isometric embedding theorem to show that simplicial polyhedra with rational edge lengths are dense in the space of simplicial polyhedra, as discussed below.

Let $\ell_i$ be the edge lengths of a given simplicial convex polyhedron $P$ and $\ell_i'$ be rational numbers with $|\ell_i'-\ell_i|\leq\epsilon$. For each face $F_i$ of $P$ let $F_i'$ be the triangle with edge lengths $\ell_{i1}',\ell_{i2}',\ell_{i3}'$, where $\ell_{i1},\ell_{i2},\ell_{i3}$ are edge lengths of $F_i$. Assuming $\epsilon$ is small, the sum of the angles of $F_i'$ around each vertex will be less than $2\pi$ if $F_i'$ are glued together the same way that $F_i$ are. Thus, by Alexandrov's theorem, gluing $F_i'$ yields a convex polyhedron $P'$. By the uniqueness part of Alexandrov's theorem, $P'$ is close to $P$, after a rigid motion, and is therefore isometric to it since $P$ is simplicial. This in turn yields that the edges of $P'$ coincide with those of $F_i'$, because edges of $F_i'$ are the unique geodesics in $P'$ connecting their end points. So $P'$ has rational edge lengths as desired.

The answer to the second question is yes for simplicial polyhedra, i.e., those with triangular faces. Indeed one may use Alexandrov's isometric embedding theorem to show that simplicial polyhedra with rational edge lengths are dense in the space of simplicial polyhedra, as discussed below.

Let $\ell_i$ be the edge lengths of a given simplicial convex polyhedron $P$ and $\ell_i'$ be rational numbers with $|\ell_i'-\ell_i|\leq\epsilon$. For each face $F_i$ of $P$ let $F_i'$ be the triangle with edge lengths $\ell_{i1}',\ell_{i2}',\ell_{i3}'$, where $\ell_{i1},\ell_{i2},\ell_{i3}$ are edge lengths of $F_i$. Assuming $\epsilon$ is small, the sum of the angles of $F_i'$ around each vertex will be less than $2\pi$ if $F_i'$ are glued together the same way that $F_i$ are. Thus, by Alexandrov's theorem, gluing $F_i'$ yields a convex polyhedron $P'$. By the uniqueness part of Alexandrov's theorem, $P'$ is close to $P$, after a rigid motion, and is therefore isometric to it since $P$ is simplicial. This in turn yields that the edges of $P'$ coincide with those of $F_i'$, because edges of $F_i'$ are the unique geodesics in $P'$ connecting their end points. So $P'$ has rational edge lengths as desired.