1.
It depends what you mean by bad. The categories still do what they are meant to do even if the underlying variety is not proper or smooth. However, there are some subtleties. For instance, if you try to pushforward a coherent sheaf along a non-proper morphism, of course the result might only be quasicoherent. Likewise, if you try to pullback a coherent sheaf from a non-regular base, then the result might be unbounded (since you might have to take an infinitely long locally free resolution). However, perfect complexes (an intrinsically defined subcategory of $D^{b}_{Coh}(X)$ behave well under pullback.
2.
Since you already have problems for pullback and pushforward, $Rf^!$ and $Rf_!$ could also not exist in non-proper or non-smooth settings. However, under fairly general hypotheses these functors will exist for the unbounded derived category of quasi-coherent sheaves. See for instance the Springer Lecture Notes of Lipman on Grothendieck duality and references therein.
3.
Gluing (recollement) can be a bit of a problem depending on the codimension of $Z$. I think it is fine for $Z$ of codimension at least $2$. (Actually, see the below comment of t3suji pointing out that codimension doesn't help.) You always have a localization sequence
$$D^{b}_{Z, Coh(X)}(X) \rightarrow D^{b}_{Coh(X)}(X) \rightarrow D^{b}(X \setminus Z).$$
However, $D^{b}_{Z, Coh(X)}(X) \rightarrow D^{b}_{Coh(X)}(X)$ might not have the desired adjoints to get a recollement and so glue. The right adjoint would be local cohomology, but these might not be coherent. ( However, you do have gluing for unbounded derived categories of quasi-coherent sheaves.
4.
I don't know the answer to this, but probably reading Toen would help. I would guess that you have to work not with triangulated categories but with their natural dg enhancements, since morphisms don't glue in the triangulated categories. That is, already for the identity morphism, I think you have problems. For instance, take a short exact sequence of vector bundles that doesn't split, so the connecting homomorphism is non-trivial. However, the sequence splits locally, so the connecting homomorphism is locally trivial.