The monoid that you are looking for is the set of birational endomorphisms of the affine plane. It is of course closed under compositions and the invertible elements are the automorphisms. You would like to study the non-trivial elements, i.e. birational endomorphisms whose inverse is not an (auto)-morphism.
The simplest example is $\pi\colon (x,y)\mapsto (xy,y)$, which contracts exactly one curve and thus cannot be a composition of two (or more) non-trivial birational endomorphisms.
The elements of the form $\alpha \circ \pi \circ \beta$, where $\alpha,\beta$ are automorphisms, are usually called simple affine contractions. In the seventies, the natural question of knowing if every birational endomorphism was a composition of such ones, or equivalently if $\pi$ generates, together with automorphisms, all the monoid of birational endomorphisms, was asked.
The answer is no and there are many counterexamples. You can for instance have a look at the article "Birational endomorphisms of the affine plane
" of Daniel Daigle: https://projecteuclid.org/download/pdf_1/euclid.kjm/1250519792
The whole monoid of birational endomorphisms is quite complicated and still mysterious now, even after a lot of nice results (the article above cites some of them, but you can find a lot more by looking on internet).