**Note**: I've appended some additional material that extends the analysis of, in effect, the pre-images of the pre-images of the fixed point $n/m=1$ of the function $f$, to look at general pre-images and pre-images of other fixed points (i.e., $n/m = (m^2-m+1)/m$ for $m>1$. The new stuff follows a boldface "Added later." **End of note**

The families Karl Fabian found can be subsumed under a single rule:

$f^{(2)}(a/b)=1$ if and only if $${a\over b}={n+d\over n+d'}$$ where
$n>0$,
$dd'=n(n-1)$, and $(n+d,n+d')=1$.

It's not hard to show that if $a/b$ has the given form, then $f(a/b)=n$, and it's easy to show that $f(n)=f(n/1)=1$. (The assumption that $n+d$ and $n+d'$ are relatively prime is crucial.) The tricky part is showing that nothing else gets to $1$ in two steps.

It's clear that $f(a/b)=1$ if and only if $a+b-1=ab$, which can be rewritten as $(a-1)(b-1)=0$, so the only numbers that get to $1$ in one step are integers $n$ and their reciprocals $1/n$. It's also clear that $f(a/b)\ge1$ for all $a/b$. Therefore, the numbers that get to $1$ in two steps are those that get to some integer $n$ in one step. Let's see how that can happen.

If $ab/(a+b-1)=n$, then we must have $ab=nk$ and $a+b-1=k$ for some integer $k$. Writing $b=nk/a$, we wind up with $a^2-(k+1)a+nk=0$, so

$$a={k+1\pm\sqrt{(k+1)^2-4nk}\over2}.$$

For $a$ to be an integer, we must have a square inside the square root:

$$(k+1)^2-4nk = m^2,$$

which can be rewritten as

$$(k+1-2n)^2-m^2 = 4n(n-1),$$

or

$$(k+1-2n+m)(k+1-2n-m)=4n(n-1).$$

The two terms on the left hand side have the same parity (they differ by $2m$), hence they must both be even, i.e., $k+1-2n+m=2d$ and $k+1-2n-m=2d'$ where $dd'=n(n-1)$. From this we get $k+1-2n=d+d'$ and $m=d-d'$, so

$$a={2c+d+d'\pm(d-d')\over2}.$$

Choosing the positive sign gives $a=n+d$. (Choosing the negative sign gives $b=n+d'$. If you like, you can assume $a\ge b$ and $d\ge d'$.)

To do just one example, let $n=16$. The choices for $dd'$ are $240\cdot1$, $80\cdot3$, $48\cdot5$, and $16\cdot15$, leading to the four possibilites for $a/b$ (with $a>b$) for which $f(a/b)=16$:

$${a\over b} = {256\over 17}, {96\over19}, {64\over21}, {32\over31}.$$

Note how a factorization like $dd'=6\cdot40$ fails:

$$f((16+6)/(16+40)) = f(22/56)=f(11/28)= {11\cdot28\over38}={154\over19}.$$

**Added later**: If I've done everything correctly, a similar analysis gives the following nice result about pre-images:

Let $n/m$ be a fraction with $n\ge m$ and
$(n,m)=1$. Then $f(a/b)=n/m$ if and
only if

$${a\over b} = {n+d\over n+d'}$$

where $dd'=n(n-m)$ and $(n+d,n+d')=m$.

Let's see how this applies to the other fixed points for $f$, namely when $n=m^2-m+1$, for the first few values of $m$.

Skipping the case $m=1$ (which you can check gives the result noted earlier), let $n/m = 3/2$, so that we need $dd'=3$ and $(3+d,3+d')=2$. The only factors are $3$ and $1$, which indeed satisfy $(6,4)=2$, but this only gives $a/b = 6/4=3/2$. In other words, the fixed point $3/2$ has no pre-image other than itself.

For $n/m=7/3$, we have $n(n-m)=28$, for which the possible factorizations are $28\cdot1$, $14\cdot2$, and $7\cdot4$. But the only one of these for which $(7+d,7+d')=3$ is $14\cdot2$, which gives $a/b = 21/9 = 7/3$, so $7/3$ also has no pre-image other than itself.

You can check that the same thing happens for $n/m = 13/4$: The only factorization $dd'$ of $13(13-4)$ for which $(13+d,13+d')=4$ is $39\cdot3$, giving $a/b=52/16 = 13/4$.

But $n/m = 21/5$, finally, is interesting: There are *two* factorizations that work, namely $84\cdot4$ and $24\cdot14$. The first, as before, gives the fixed point again, $$a/b = (21+84)/(21+4) = 105/25 = 21/5.$$ But the other one gives $$a/b = (21+24)/(21+14) = 45/35 = 9/7.$$ Note, however, that $9/7$ has *no* pre-image: The factorizations of $9(9-7)=18$ are $18\cdot1$, $9\cdot2$, and $6\cdot3$, none of which produce anything divisible by $7$, much less a pair $(9+d,9+d')$ with $7$ as a common divisor.

In summary (for now), the only fraction with infinitely many pre-images is $n/m=1$ (since $n(n-m)=0$ has infinitely many divisor pairs!); the size of the pre-image set for all other fractions is bounded by the number of divisor pairs of $n(n-m)$. For some of the fixed points of $f$, the pre-image set is just the fixed point itself, while for others (e.g. $n/m = 21/5$), the pre-image set contains additional points. There may be some simple criterion that identifies the fixed points that have no additional pre-images, but I don't offhand see one, possibly because I haven't thought hard enough about it (but maybe because I'm just blind to the obvious).

11more comments