This problem is likely best described with the following picture:
Given the discrete set of points $A$ (shown in blue), I wish to calculate the discrete set of points that are contained within the convex hull of set $A$, producing set $B$ (shown in red). Note that it is not necessarily required to calculate the convex hull itself, although it may be advantageous to do so.
I have been able to achieve this by testing whether each point within the bounding box (shown in black) is linearly separable from the entire set $A$ (using linear programming). However this is very slow when set A is large and in higher dimensions and I feel like testing each point is not necessary.
Preferably I'd like some advice on where to look for optimizing my search for points. Maybe fit a rectangle inside to avoid testing all points? I'm just looking for ideas or other solved methods.
For my application I need to extend this to 3D but I find solving these problems in 2D and generalizing much easier.