Our paper specializes to the case of 1-categories with (co)limits and (co)limit-preserving functors, not to 2-categories.
But surely this is true. I'm defining the comma object in the 2-category of (strict) 2-categories, 2-functors, and 2-natural transformations. The defining universal property of $F \downarrow G$ characterizes maps $A \to F\downarrow G$. Considering $A$ to be terminal, the walking arrow, and the walking 2-cell tells us how to define the 2-category $F\downarrow G$. For instance: a 2-cell is a 2-cell in $C$ and a 2-cell in $B$ together with a 2-natural transformation from the image of the former to the image of the latter. The data of this 2-natural transformation consists of a pair of maps in $D$, which is part of the data of the pair of objects serving as domain and codomain of this 2-cell. Note that the underlying category of this 2-category is the comma category of the underlying categories.
Now a cospan in $F \downarrow G$ is just a cospan in the underlying comma category. Form the comma objects in $C$ and $B$. Their images in $D$ are still comma objects and the legs of the cospan together with the 2-cell of the former comma object defines a cone over the latter, inducing a unique map between the comma objects so that everything commutes. This defines the comma object in $F \downarrow G$ and the universal property is proven similarly.
I'd suspect, guided by Blackwell-Kelly-Power's "Two dimensional monad theory" that any PIE-limit (including pseudo-limits and lax-limits) of 2-categories with and 2-functors that preserve certain 2-(co)limits will have those 2-(co)limits and they will be preserved by the legs of the limit cone. But I don't know whether such 2-categories can be characterized as algebras for a 2-monad. The google preview of Power's "Three dimensional monad theory" suggests that a Gray-monad is needed. And I don't know whether anyone has proven the corresponding completeness result in that context. You might ask the categories list?