This is a great question for the theory of dynamics of polygon iterations. The best-known map of this kind is undoubtedly the pentagram map. The pentagram map sends a pentagon to the pentagon formed by the five diagonals; Wikipedia explains it best. Richard Schwartz proved that there is a limit point of the kind you describe if the starting pentagon is strictly convex. Amazingly, Max Glick proved the coordinates of that limit point are algebraic functions (using at worst cube roots) of the original pentagon's coordinates. Further studies were made by Aboud-Izosimov that give a kind of theoretical explanation for why that limit point exists.
Questions like this can be difficult and are very sensitive to the particular operations you use. For instance, unlike the pentagram map, your map does not preserve convexity. Neither is it a rational map – the coordinates of the image introduce square roots in general. All this makes geometric intuition difficult to apply. All this is to say, to my eye this is a worthwhile research problem that shouldn't have an out-of-the-box answer from general theory. Here are two questions to get you thinking:
Why not start with quadrilaterals, or even triangles?
If you draw the "exterior" angle bisector, that is, the perpendicular to the angle bisector, it looks like you get a convexity-preserving map. Do these tend to regular polygons up to resizing?
References:
Glick, Max, The limit point of the pentagram map, Int. Math. Res. Not. 2020, No. 9, 2818-2831 (2020). ZBL1484.37075.
Aboud, Quinton; Izosimov, Anton, The limit point of the pentagram map and infinitesimal monodromy, Int. Math. Res. Not. 2022, No. 7, 5383-5397 (2022). ZBL1486.52001.
Schwartz, Richard, The pentagram map, Exp. Math. 1, No. 1, 71-81 (1992). ZBL0765.52004.