I want to deform a 3D mesh according to 3 or more control points, meaning that the transformation is constituted by pre-images $c_i$ and images $c_i'$ of these control points. Each point of the mesh should be smoothly interpolated, so that, if a point of the mesh equals a control point $c_i$, it should be mapped to $c_i'$. Beyond the convex hull of the control points the transformation should be carried on radially, but also smoothly around the corners (I don’t know how to describe this better). I thought about inversely weighting the influence of each control point by the distance, but I don’t know how to do this in detail. If there are only three control points the transformation should only consist of translation, rotation, scale and shear mappings. Can you give me some hints what kind of mathematical tools are available to do this?
Deformations of meshes is a heavily studied topic. Perhaps looking at this 2012 survey will help focus your investigation:
"A Comparison of Mesh Morphing Methods for 3D Shape Optimization." Matthew L. Staten, Steven J. Owen, Suzanne M. Shontz, Andrew G. Salinger, and Todd S. Coey. Proceedings of the 20th International Meshing Roundtable, 2012, pp 293-311. (Springer link)