This is a great question for the theory of dynamics of polygon iterations. The best-known map of this kind is undoubtedly the pentagram map. The pentagram map sends a pentagon to the pentagon formed by the five diagonals; Wikipedia explains it best. Richard Schwartz proved that there is a limit point of the kind you describe if the starting pentagon is strictly convex. Amazingly, Max Glick proved the coordinates of that limit point are algebraic functions (using at worst cube roots) of the original pentagon's coordinates. Further studies were made by Aboud-Izosimov that give a kind of theoretical explanation for why that limit point exists.

Questions like this can be difficult and are very sensitive to the particular operations you use. For instance, unlike the pentagram map, your map does not preserve convexity. Neither is it a rational map – the coordinates of the image introduce square roots in general. All this makes geometric intuition difficult to apply. All this is to say, to my eye this is a worthwhile research problem that shouldn't have an out-of-the-box answer from general theory. Here are two questions to get you thinking:

1. Why not start with quadrilaterals, or even triangles?

2. If you draw the "exterior" angle bisector, that is, the perpendicular to the angle bisector, it looks like you get a convexity-preserving map. Do these tend to regular polygons up to resizing?

References:

Glick, Max, The limit point of the pentagram map, Int. Math. Res. Not. 2020, No. 9, 2818-2831 (2020). ZBL1484.37075.

Aboud, Quinton; Izosimov, Anton, The limit point of the pentagram map and infinitesimal monodromy, Int. Math. Res. Not. 2022, No. 7, 5383-5397 (2022). ZBL1486.52001.

Schwartz, Richard, The pentagram map, Exp. Math. 1, No. 1, 71-81 (1992). ZBL0765.52004.

The operation you are describing is called the "discrete evolute" of a plane polygon. it turns out that the first and third steps are homothetic (this essentially means equivalent up to scaling). Iteration is then a matter of linear algebra. This should provide you a route to proving the kind of convergence you describe: write down the matrix that carries out the homothety in coordinates, and look for a real eigenvector with an attracting eigenvalue. The results you need should be in Section 4.4.3 of the linked paper.

Arnold, M., Fuchs, D., Izmestiev, I. et al. Iterating Evolutes and Involutes. Discrete Comput Geom 58, 80–143 (2017). https://doi-org.ezp-prod1.hul.harvard.edu/10.1007/s00454-017-9890-y

-- Previous answer --

This is a great question for the theory of dynamics of polygon iterations. The best-known map of this kind is undoubtedly the pentagram map. The pentagram map sends a pentagon to the pentagon formed by the five diagonals; Wikipedia explains it best. Richard Schwartz proved that there is a limit point of the kind you describe if the starting pentagon is strictly convex. Amazingly, Max Glick proved the coordinates of that limit point are algebraic functions (using at worst cube roots) of the original pentagon's coordinates. Further studies were made by Aboud-Izosimov that give a kind of theoretical explanation for why that limit point exists.

Questions like this can be difficult and are very sensitive to the particular operations you use. For instance, unlike the pentagram map, your map does not preserve convexity. Neither is it a rational map – the coordinates of the image introduce square roots in general. All this makes geometric intuition difficult to apply. All this is to say, to my eye this is a worthwhile research problem that shouldn't have an out-of-the-box answer from general theory. Here are two questions to get you thinking:

1. Why not start with quadrilaterals, or even triangles?

2. If you draw the "exterior" angle bisector, that is, the perpendicular to the angle bisector, it looks like you get a convexity-preserving map. Do these tend to regular polygons up to resizing?

References:

Glick, Max, The limit point of the pentagram map, Int. Math. Res. Not. 2020, No. 9, 2818-2831 (2020). ZBL1484.37075.

Aboud, Quinton; Izosimov, Anton, The limit point of the pentagram map and infinitesimal monodromy, Int. Math. Res. Not. 2022, No. 7, 5383-5397 (2022). ZBL1486.52001.

Schwartz, Richard, The pentagram map, Exp. Math. 1, No. 1, 71-81 (1992). ZBL0765.52004.