Do we have specific examples of h-cobordant smooth manifolds $M$ and $M'$ such that $\operatorname{BDiff}(M) \not \simeq \operatorname{BDiff}(M')$? Perhaps something can be said in terms of K-theory by analyzing bundles of h-cobordisms?
In fact, if it helps to stabilize, I would be happy to know that $\operatorname{BDiff}_{\partial}(M \times I)) \not \simeq \operatorname{BDiff}_{\partial}(M’ \times I))$$\operatorname{BDiff}_{\partial}(M \times I) \not \simeq \operatorname{BDiff}_{\partial}(M’ \times I)$.