The "rule" for the generalized linear fractional transformation in these coordinates is simply the re-Iwasawa-decomposition of $g\cdot n(x)a(y)$.

I think it is just at this point that the writing of $x,y$ as $z=x+*y$ becomes much less tenable. Indeed, for hyperbolic $n$-space as $SO(n,1)/O(n)$ there seems to be no useful such expression. (Of course, this does not exclude expressing such things in terms of various associated Clifford algebras, but the happy coincidences that occur in the very small cases cease.)

If that $I(gp)$ is meant to be the analogue of "imaginary part", i.e., the split-Levi part $y$, then you can obtain the formula for it from the re-Iwasawa-decomposition. The $O(n,1)/SO(n)$ model for hyperbolic $n$-space still does admit a reasonable expression, as well.