Are there pre-existing algorithms that will determine maximum number of possible polygons (rectangles only, or triangle only) that can "fill" a non regular area in 2D.
I think your question needs sharpening. Perhaps rather than maximum you mean minimum? And what does it mean to "fill" a "non regular area"? Partition? Cover?
If you are interested in partitioning a polygon into triangles, then see the Wikipedia article on "polygon triangulation." If you are interested in covering a polygon with rectangles, then perhaps start with the 1990 paper, "Covering rectilinear polygons by rectangles," IEEE link here. If instead you want to cover with squares, see the earlier MO question, "Covering an arbitrary polygon with minimum number of squares."