First of all, note that if $\lambda$ is a singular limit cardinal, then the forcing is terribly behaved already in $\sf ZFC$. Even if $\lambda$ is a successor, you'd still find that the same argument works. Divide $\lambda$ into a small amount of intervals, since any small set, cardinality wise, must be bounded in almost all of them, we are guaranteed that generically the collapsing function is constant on a tail of cofinally many of the intervals, which gives us a surjection from $\operatorname{cf}(\lambda)$ onto $\kappa$, and therefore you've collapsed much more that you've bargained for.
In the more general case. Forcing with "plain ol' sequences of ordinals" is a bad idea when working in $\sf ZF$, and will rarely work. The reason is simple, if we are just concatenating sequences of ordinals without any restrictions, we are pretty much guaranteed to introduce some well-orderings into the universe. This is very easy to see in the case of $\kappa=\lambda$ which is just adding a Cohen subset to $\lambda$, in which case we force a bijection between $\mathcal P(\lambda)^V$ and $\lambda^+$, which will add well-orderings for $\mathcal P(\alpha)^V$ below $\lambda$ as well, and in that case bounded subsets could be added below $\lambda$.
More to the point, if $\kappa^{<\lambda}$ can be well-ordered, then as a well-ordered forcing things are more or less reasonably behaved. So, for example, if we assume $\sf AX_4$, which states that for any infinite ordinal $\kappa$, $[\kappa]^{\omega}$ is well-orderable. So, in this case $\operatorname{Col}(\omega_1,\kappa)$ is going to be a reasonably behaved forcing.
In the general case, it's not at all clear that there is a way in $\sf ZF$ to add generic sets of ordinals without "damage" (with the exception of Cohen reals, or more generally, $\operatorname{Col}(\omega,\alpha)$ which is likely to not cause more damage than intended). This is more of a situation that will need to be tailored to your current model. For example, if you're working in a symmetric extension, ground model forcings are well-orderable and will commute with the symmetric extension, so they can sometimes be useful and preserve nice things (but not always!). I have always seen that as one of the biggest problems in the theory of forcing without choice, and we have yet to even discuss non well-orderable sets.