My purpose is to verify an historical hypothesis I have on Gauss's tesselation of the unit disk as described in John Stilwell "Mathematics and its history". Looking at the relevant pages in Gauss's Nachlass (volume 8, p.102-105), I read that the commentor (Robert Fricke) on this fragment of Gauss says that Gauss's drawing (the (4 4 4) tesselation) is intended to be a geometrical illustration for composition of substitutions other then the fundamental generators of the modular group. The following sentences are a citation of Fricke about the substitutions Gauss used:
Gauss has repeatedly dealt with composition of other substitutions of the group defined from these generators. In addition to the information in fragment [i], the following formula should also be mentioned $$\frac{[\alpha, \beta,\dots,v]\theta + [\beta,\gamma,\dots,v]i}{-i[\alpha,\beta,\dots,\mu]\theta+[\beta,\gamma,\dots,\mu]}$$ which can be found in a booklet entitled “Cereri Palladi Junoni sacrum, Febr. 1805”. The continued fraction expansions of the two substitutions are given as examples:$$\frac{128\theta + 37i}{-45i\theta + 13}$$ $$ \frac{121\theta+36i}{-84i\theta+25}$$
Just to explain the words of Fricke, "the group defined from these generators" is the modular group (since the generators mentioned before this sentence are addition of $i$ and inversion with respect to the unit circle), and the $[,,\dots,]$ is the "Gaussian Brackets" notation (which Gauss defined in article 27 of Disquisitions Arithmeticae, and is connected directly to continued fractions). I think $\theta$ is just a notation for the complex variable that is transformed under the Mobius transformation.
Checking the determinants of these substitutions gave $-1$ for the first one and $+1$ for the second one, so this made me suspect that these are isometric Mobius transformations (the only thing that doesn't settle is that $a,b,c,d$ in the Mobius transformation should be real integers, not imaginary integers). One should also notice that the second Mobius transformation can be rewritten as:
$$\frac{121\theta+36i}{-84i\theta+25} = \frac{[3,2,1,3,3]\theta+[2,1,3,3]i}{-[3,2,1,3,2]i\theta+[2,1,3,2]}$$
, in exact accordance with Gauss's formula which uses the Gaussian Brackets notation (here $\alpha=3, \beta=2,\gamma=1, v=3, \mu=2$).
In addition, the diagonal elements of the second Mobius transformation ($121$ and $25$) are both reduced to $1$ modulo $12$, while the off-diagonal elements reduce to $0$ modulo $12$ (if one allows imaginary sizes for $b,c$). In other words, this shows that the second Mobius transform belongs to a congruence subgroup of level 12 in $SL_{2}(Z[i])$.
Although it seems at first that these Mobius transformations are just examples of a general principle of certain continued fractions developement, immediately after that Fricke says:
Both to explain the continued fraction development of the substitutions and to draw conclusions from the theory of functions, Gauss made use of the geometrical representation which has become the basis of the more recent theory of module functions. In the booklet just mentioned, Gauss drew the figure shown here. Since the above-mentioned continued fraction expansions of substitutions are also to be found, Gauss must have used the figure as a means to illustrate these continued fraction expansions. In fact, one has here the beginning of the well-known network of circular arc triangles, which is the basis of the theory of modulus functions. It's evident that Gauss generally understood the "principle of the symmetrical multiplication of curved triangles", which comes into consideration here, and even the character of the "natural limit" of a triangular network to be obtained in this way did not remain hidden... These are circular arc triangles of the angles $\frac{\pi}{4},\frac{\pi}{4},\frac{\pi}{4}$, and the orthogonal circle highlighted in the drawing represents their natural limit. In addition to the drawing, the following information was written by Gauss: "Center of the first circle: $2^{\frac{1}{4}}$, radius of the first circle: $\sqrt{\sqrt{2}-1}$, center of the second circle: $\frac{1}{2}(\sqrt{\sqrt{2}+1}+\sqrt{\sqrt{2}-1})$, radius of the second circle:$\frac{1}{2}(\sqrt{\sqrt{2}+1}-\sqrt{\sqrt{2}-1})$.
Therefore, it seems that these Mobius substitutions are actually the generators for some tiling of the hyperbolic disk. But this conclusion is a result of a very shallow reading of Fricke's comments and I lack the proffesional knowledge needed to verify my reading.
I also found another reference to Gauss's hyperbolic tiling in p.198 of Schlesinger's essay on Gauss's researches in function theory, where he says:
The extent to which Gauss must also be regarded as a precursor to the theory of automorphic functions is shown not only by his construction of the fundamental domain for the modular function, but also by his interpretation of the linear fractional substitution $$t'=\frac{(\alpha +\beta i)t-(\gamma + \delta i)}{(\gamma - \delta i)t+(\alpha-\beta i)}$$ as a rotation of the spherical surface in itself, which is discussed in two notes printed in volume 8, p.354-356 the works, and that relates to a case of transcendent triangular functions that do not correspond to the modular functions, a polygonal network, found in the p.104 of volume 8 of the works, after a statement on a seperate page.
In the introduction to the book "Papers on Fuchsian Functions" (which is available on internet archive), John Stillwell says several things that seem to me very helpful:
Triangle functions arise in connection with the hypergeometric differential equation of Gauss, as Gauss himself seems to have discovered. This can be guessed from Figure 3, which is an undated fragment found in his Nachlass (see Gauss's werke, volume 8, p. 102-105). The figure is a tessellation of the unit disk by curvilinear triangles with angles $\pi/4$, and it reflects the periodicity of the inverse to a certain quotient of solutions to the hypergeometric equation. The group in this case consists of certain linear fractional transformations which leave the unit disc invariant. Invariance under linear fractional transformations arises quite generally from second order linear differential equations as follows. If $y_1,y_2$ are two linearly independent solutions of $$\frac{d^2y}{dx^2}+p(x)\frac{dy}{dx}+q(x)=0$$ then any other solutions $y'_1,y'_2$ are linear combinations of $y_1,y_2$: $$y'_1 = ay_1+by_2$$ $$y'_2 = cy_1+dy_2$$ hence the quotient of the new solutions is related to the quotient of the old by a linear fractional transformation: $$\frac{y'_1}{y'_2} = \frac{a\eta+b}{c\eta+d}$$ where $\eta = \frac{y_1}{y_2}$. By inverting the quotient of solutions one obtains a function automorphic with respect to a certian group of linear transformations. In the case of the hypergeometric equations, one is led to groups of automorphisms of certain triangle tessellations of the unit disc. The Gauss's figure is one such tessellation. Others were found by Riemann in lectures of 1858-1859 (discovered only in 1897 and published in Riemann [1902]), and the general theory was worked out independently by Schwarz [1872].
I don't now on what sources Stillwell bases his conjecture about the origin of Gauss's tessellation. After I asked people at the department for handwritten manuscripts and rare prints of the Gottingen university to search for the location of Gauss's drawing, they managed to find it after many efforts. Gauss's drawing (together with his related notes on the metrical relations) from p.104 of volume 8 of his works appears in an obscure notebook called Cod. Ms. Gauss Varia 2, as can be seen from this link.
After gaining access to this notebook, I saw only one more drawing in it related to this tessellation - but again, with no supporting context.
Therefore, I believe my attempts to answer that question arrived into a deadlock, and I don't understand how Stillwell came with his "hypergeometric function conjecture" about the origin of this tessellation. However, I still believe his notes on the mertical relation exhibit knowledge of the Poincare disk model (simply because I cannot explain their correctness in other ways).
Historical significance of Gauss's results:
Being the first drawing of it's kind, the tessellation drawed by Gauss and his related results have planted some the seeds of Felix Klein's "Erlangen program" (with the other influences being Galois's theory of equations and Riemann's geometric ideas). Klein read Gauss's fragments very closely and seems to have been influenced by them, so I think it's not an exaggeration to say that Gauss's drawing was one of his sources of inspiration.
Therefore I believe that for a correct historic appreciation of the roots of Erlangen program, it's important to know how Gauss arrived at the metrical relations in his tiling (the results stated by Gauss on the location and radiuses of the centres of the first and secondary circles in his tessellation are already confirmed by my posted answer - although it might not be the original method of Gauss).
Questions
Do Gauss's formulas for the radiuses of curvature of the first and second generation circles imply that he essentialy knew the Poincare disk model of the hyperbolic plane? I ask that because the derivation presented makes heavy use of the conception and machinery of this model. I find it unprobable, at least according to traditional accounts of Gauss's mathematical interests (the disk model is essentially a conformal model of the hyperbolic plane, and Gauss didn't think about conformal mappings until at least 1815).
It's more likely that he arrived at this result in the context of his analytic investigations, and not in the context of his "meditations" on non-Euclidean geometry.