Gauss's procedure leads to Bolyai's result on the volume of orthoscheme tetrahedron, as I'll show here. However, Gauss's result is a little bit more limited than Bolyai, since Gauss refers to an orthoscheme tetrahedron of which 4 of the 12 face angles of the tetrahedron are right (each face is an hyperbolic right triangle), while Bolyai refers to a slightly more general tetrahedron whose only 3 face angles ar right.
In order to help visualize the relations, I added here a pic of Gauss's note.
Preliminary discussion:
Relation to Schlafli formula
To see the connection between Schlafli formula and the first formula in Gauss's fragment, one needs to understand that Gauss thinks of the tetrahedron 1234 in such a way that the faces 124 and 134 are perpendicular and the edges 24 and 13 meet the intersection line 14 also at right angles. So if we draw a unit sphere centered at vertex 1, we generate a right spherical triangle 234 (this spherical triangle has right angle at vertex 4 - this angle is simply the dihedral angle 14) and side length 34 which is also equal to a right angle arc; hence, vertex 3 is the pole of arc 24 and therefore the angle at vertex 2 (which is dihedral angle 12) is also right. Therefore, both dihedral angles 14 and 12 don't contribute to the sum in Schlafli formula.
In addition, Gauss defines the tetrahedron in such a way that the angles at vertex 3 are constant (so that an "observer" in hyperbolic space which is located at vertex 3 sees the rest of the vertexes at constant lines of sight). Since the three face angles at vertex 3 correspond to the length of sides of a spherical triangle, and the dihedral angles at sides 31,32,34 correspond to the angles of this spherical triangle, one gets that constancy of face angles at vertex 3 implies constancy of the dihedral angles 31,32,34.
Therefore, only the dihedral angle of the side 24 changes. By repeating the same argument on a unit sphere (argument which we used on vertex 1) centered at vertex 4, we conclude that vertex 2 of the spherical triangle 123 is the pole of side 13 so two face angles at vertex 4 are right (face angles 142 and 342) and dihedral angle 24 is equal to face angle 341. All this leads directly to the first formula in Gauss's fragment (apart from a missing factor of $\frac {1}{2}$).
Derivation of the fundamental metrical relation for an orthoscheme
To derive the metrical relation $\alpha^2 \mathbb{cot}^2(341)-\beta^2\mathbb{tanh}^2(24) = 1$, we use the fact that all faces of the tetrahedron are hyperbolic right triangles. For the right hyperbolic triangle 134 we have:
$$\mathbb{cosh}(34) = \mathbb{cot}(431)\mathbb{cot}(341)$$
and for the right hyperbolic triangle 234 we have:
$$\mathbb{sinh}(34) = \mathbb{cot}(234)\mathbb{tanh}(24)$$
so therefore we have the equation:
$$\mathbb{cot}^2(431)\mathbb{cot}^2(341) - \mathbb{cot}^2(234)\mathbb{tanh}^2(24) = 1$$
Derivation of explicit volume formula from Gauss's formula:
For the sake of consistency, we denote the angles 431, 234, and 214 as $\alpha$, $\beta$ and $\gamma$, respectively. Now lets look at the link of vertex 3 of the tetrahedron: it is a spherical triangle whose two edges lengths are $\alpha$ and $\beta$ and one angle is $\gamma$ (it is the dihedral angle of edge 31 and it is also equal to $\gamma$). In addition the sides $\alpha$$, \beta$ of this spherical triangle are orthogonal to each other. Therefore, by a combination of the spherical sine theorem and the spherical pythagoras theorem, we get:
$$ \frac{{\mathbb{sin}(\mathbb{arccos}(\mathbb{cos}\alpha\cdot \mathbb{cos}\beta))}}{{\mathbb{sin} (\frac{\pi}{2}})} = \frac {{\mathbb{sin}\beta}}{{\mathbb{sin}\gamma}}$$, or:
$$(1) \mathbb{sin}\gamma = \frac {{\mathbb{sin}\beta}}{{\sqrt{{1 - (\mathbb{cos}\alpha \cdot \mathbb{cos}\beta)^2}}}}$$
Now, denote the length of side $24$ as $l_{24} = x$ and the angle $341$ as $\varphi$. Since $\varphi$ is related to $x$ by the equation $c^2_1 \mathbb{cot}^2\varphi - c^2_2\mathbb{tanh}^2x = 1$ (here $c_1 = \mathbb{cot}\alpha,c_2 = \mathbb{cot}\beta$), one can write:
$$\varphi = \mathbb{arccot}(\frac{\sqrt{1+c^2_2\mathbb{tanh}^2x}}{c_1})$$
Gauss's procedure for the calculation of the volume, which uses the relation $\partial \Delta = -\frac{1}{2}x d\varphi$, leads to the following integral:
$$\Delta = -\frac{1}{2}\int x d\varphi = -\frac{1}{2}\int x \frac{d\varphi}{dx}dx$$
so one can compute the derivative of $\varphi$ with respect to $x$ by an application of the chain rule:
$$-\frac{d\varphi}{dx} = \frac{1}{1+\frac{1+c^2_2\mathbb{tanh}^2x}{c^2_1}}\frac{c^2_2\mathbb{tanh}x\cdot \frac{1}{\mathbb{cosh}^2x}}{c_1\sqrt{1+c^2_2\mathbb{tanh}^2x}}$$
Now make a very long algebraic simplification:
$$ -\frac{d\varphi}{dx} = \frac{\mathbb{sinh}x (c^2_2/c_1)}{\mathbb{cosh}^2x(1+\frac{1+c^2_2\mathbb{tanh}^2x}{c^2_1})\mathbb{cosh}x\cdot c_2\sqrt{\frac{1}{c^2_2}+\mathbb{tanh}^2x}} = \frac{\mathbb{sinh}x (c_2/c_1)}{\mathbb{cosh}^2x(1+\frac{1+c^2_2\mathbb{tanh}^2x}{c^2_1})\sqrt{\frac{\mathbb{cosh}^2x}{c^2_2}+\mathbb{sinh}^2x}} = \frac{\mathbb{sinh}x (c_1/c_2)}{(c_1/c_2)^2(\mathbb{cosh}^2x(1+\frac{1+c^2_2\mathbb{tanh}^2x}{c^2_1}))\sqrt{\frac{\mathbb{cosh}^2x}{c^2_2}+\mathbb{sinh}^2x}}$$
The expression under the square root is $$\sqrt{\frac{\mathbb{cosh}^2x}{c^2_2}+(\mathbb{cosh}^2x-1)} = \sqrt{\mathbb{cosh}^2x(\frac{1}{c^2_2}+1)-1} = \sqrt{{\frac{{\mathbb{cosh}^2x}}{{\mathbb{cos}^2\beta}} - 1}}$$
while the expression in the left side of the denominator is equal to:
$$(c_1/c_2)^2\mathbb{cosh}^2x(1+\frac{1}{c^2_1})+\mathbb{sinh}^2x = \mathbb{cosh}^2x(1+\frac{c^2_1+1}{c^2_2}) -1 = \mathbb{cosh}^2x(1+\frac{1}{\mathbb{sin}^2\alpha \mathbb{cot}^2\beta})-1$$
Recalling that $\frac{c_1}{c_2} = \frac {\mathbb{tan}\beta}{\mathbb{tan} \alpha}$, the resulting expression for the integral is:
$$\Delta = \frac {{\mathbb{tan}\beta}}{{2 \mathbb{tan} \alpha}}\int_{0}^{c}\frac {{x \mathbb{sinh}(x) dx}}{{(\mathbb{cosh}^2x(1 + \frac {{1}}{{\mathbb{sin}^2\alpha \mathbb{cot}^2\beta}})-1)\sqrt{{\frac{{\mathbb{cosh}^2x}}{{\mathbb{cos}^2\beta}} - 1}} }}$$
Now, the left factor of the denominator $\mathbb{cosh}^2(x)(1 + \frac{{1}}{{\mathbb{sin}^2\alpha \ \mathbb{cot}^2\beta}})-1$, is exactly equal to $\mathbb{cosh}^2(x)\cdot \frac{{1}}{{\mathbb{cos}^2\gamma}}-1$, because subtitution of $\mathbb{cos}\gamma = \sqrt {1 - \frac {{\mathbb{sin}^2\beta}}{{1-(\mathbb{cos}\alpha\cdot \mathbb{cos}\beta)^2}}}$
(this substitution is true because of relation (1)) in this expression gives the previous one.
Concluding remarks:
- As can be seen from this presentation - Hyperbolic Volumes and Symmetry, the Bolyai's volume integral is written in my notation in this way (see Theorem 5, p. 12, at this presentation) :
$$Vol(T) = \frac {{\mathbb{tan}\beta}}{{2 \mathbb{tan} \alpha}}\int_{0}^{c}\frac {{x \mathbb{sinh}(x) dx}}{{(\frac {{\mathbb{cosh}^2(x)}}{{\mathbb{cos}^2\gamma}} - 1)\sqrt{{\frac{{\mathbb{cosh}^2x}}{{\mathbb{cos}^2\beta}} - 1}} }}$$
and in the case treated here Bolyai's integral coincides with the result of Gauss's procedure. Important Note: the differences in notation between the Bolyai integral in the presentation and Gauss's integral are just due the different symbols of the angles 431, 234, and 214 - $\alpha,\beta,\gamma$ in the presentation correspond to $\gamma, \alpha ,\beta$ in my notation.
However, for the case treated by Gauss, his formulas are absolutely correct. He should also be given credit for the identification of the calculation of the orthoscheme tetrahedron as the basis for volume formulas of general tetrahedrons (without right angles). In one of his letters, he refered to those calculations of volumes as "die jungle" - I guess he refered to the extremely complicated integrals that arise in the attempts to the decompose the general tetrahedron into orthoscemes (this problem was only solved very recently).
- Paul Stackel, the mathematician who edited Janos Bolyai's geometric works, had the following things to say about Bolyai's derivation of his integral formula:
It is most remarkable that the method that Gauss used for cubing the tetrahedron, is exactly the same as that of
Johann. This is shown in a note from March 1832, from
Gauss's estate, which is printed in the works (vol. VIII, p. 228);
Gauss has exactly the same special tetrahedron (only 3142 instead
$abc\delta$ means) and exactly the same decomposition by planes perpendicular to ab (31).
This quotation is taken from p. 113 of the book "Wolfgang und Johann Bolyai geometrische Untersuchungen" (here is a link: https://archive.org/details/wolfgangundjohan01stuoft/page/112/mode/2up), which was edited and translated to german by the Stackel.
- It's still necessary to understand how Gauss arrived at the formula $\partial \Delta = -\frac{{1}}{{2}}(24)d(341)$ (he missed the factor $\frac {{1}}{{2}}$ at the first attempt). In his commentary on Gauss's note, Stackel derives it in the following way:
The tetrahedron $1234$, whose volume is called $\Delta$, may now experience an infinitely small increase in volume $1 2 4 1' 2' 4' = \partial \Delta$, by lengthening the edge $31$ by the infinitely small amount $11' = d(13)$ and through $1'$ a perpendicular to $31'$ laying in the plane $1'2'4'$, which intersects the edges $32$ and $34$ in $2'$ and $4'$, respectively. The angles at the corner $3$, and thus also the sizes $\alpha$ and $\beta$, remain unchanged. The angle $(341)$ changes into the angle $$(34'1') = (3 4 1)+d(3 4 1)$$ namely, like the consideration of the quadrilateral $1 1' 4' 4$ with the infinitely small base line $1 1' = d(1 3)$ and right angles at $1$ and $1'$ recognized immediately: $$d(3 4 1) = \mathbb{sinh}(14)\cdot d(13)$$ The increase in volume $1 2 4 1' 2' 4'$ is bounded laterally by the triangles $1 2 4$ and $1' 2' 4'$, whose planes are both perpendicular to $1 1'$, and hence (see p.233 of this volume): $$\partial \Delta = -\frac{1}{2}d(13)\cdot(24)\mathbb{sinh}(14)$$ hence:$$\partial \Delta = -\frac {1}{2}(24)\cdot d(3 4 1) $$ and that is, apart from the missing factor $\frac{1}{2}$, Gauss's formula.
Since Stackel refers to Gauss's second fragment on volume determinations in non-euclidean geometry (p. 233 of he same volume), which was written in 1840 and was found next to Gauss's copy of one of Lobachevski's publications, I think understanding Gauss's second fragment may help understanding Gauss's reasoning.
Gauss's second fragment on volume determinations in hyperbolic geometry
It can be translated as:
$Aa,Bb,Cc$ are normals to the plane containing the triangle $ABC$. Through $a,b,c$ a second plane. $Aa$ is the smallest distance between the two planes, and itself infinitely small. $AD$ normal to $BC$. Then:$$Volume\space
ABCabc = \frac{1}{2}Aa\cdot BC \cdot \mathbb{sinh}(AD)$$
Even more delicate: volume of the pyramid $ABCD$ in which the angle $ABC$ = 90 degrees and the infinitely small dihedral angle at $AB=\theta$, $$AB = x,$$ $$AC = r.$$ $$Volumen = \frac{1}{2}(x-r\mathbb{cos}\varphi)\theta$$
(Here $\varphi$ is the angle of face $ABC$ at corner $A$, as is clear from Gauss's drawing (not shown here).)
In the first part of this fragment, the (infinitesimal) volume element that Gauss gives a formula for is an hyperbolic wedge, and according to p.114-115 of the book "Geometry II: Spaces of Constant Curvature", this result is indeed correct. There, the author (E.B. Vinberg) proves the result:
The volume of a hyperbolic wedge is equal to
$$\frac{1}{n-1}(Edge\space volume)(Slice\space area)$$
The author shows that this formula becomes $\frac{1}{n-1}\mathbb{sinh}(a)Vol (dP)dt$, where $dt$ is the thickness of the wedge ($Aa$ in Gauss's notation), and $a$ ($AD$ in Gauss's notation) is the normal to edge $dP$ ($BC$ in Gauss's notation). For three dimensions ($n=3$) this is exactly Gauss's formula.
Regarding the second part of Gauss's fragment (about the pyramid $ABCD$) - the formula Gauss gives is consistent with Schlafli formula. To see this, note that if the dihedral angle $\theta$ is very small, than varying it changes only dihedral angles at sides $AB,AC,AD$ of the pyramid. The dihedral angles at sides $AC,AD$ are equal (we will denote it $\beta$), and if we increase $\theta$ than $\beta$ decreases by $\frac{1}{2}\mathbb{cos}\varphi d\theta$. To see why this statement is correct all we need is to consider the link of vertex $A$, which is a spherical triangle with two sides $\varphi$ that enclose an infinitesimal angle $\theta$. Than a simple calculation shows that $\beta-\frac{\pi}{2}$ - which is the change in dihedral angles $AC,AD$ - is $-\frac{1}{2}\theta\mathbb{cos}\varphi$. Now combine this with the fact that $$AB=x$$
$$AC=AD=r$$
and use Schlafli formula to get Gauss's result:
$$Volumen = \frac{1}{2}(x-r\mathbb{cos}\varphi)\theta$$
Concluding remarks
According to one of the letters in the Bolyai-Gauss correspondence, after Gauss recommended Bolyai to determine the volume of hyperbolic tetrahedron, Bolyai's father replied that his son has already dealt with such volume calculations before. However, Since Bolyai's Appendix does not contain any volume calculation, Gauss's first fragment must be a result of original thought process of him. On the other hand, as noted by Stackel, his first fragment does not contain a factor of $\frac{1}{2}$, and this made me wonder if he was truly aware of Schlafli's formula in its full generality (at least in three dimensions). To be sure, his procedure shares some striking similarities with it, like the use of volume differential $\partial \Delta$ (his idea treats a tetrahedron as part of parametric family, instead of trying to calculate volume directly from length of sides) and of the dihedral angle at side $24$ (use which is somewhat hidden by the fact that it equals angle $341$).
Certainly, Gauss later corrected the mistake, but this was apparently only after he became acquainted with Lobachevsky's work. According to Stackel's comments, the second formula in Gauss's second fragment appears as formula 62 in Lobachevsky's work "On the Principles of Geometry" (1830), which can be found here. Immediately after that (in [43]-[49] of this work), Lobachevsky communicates his discovery of an expression for volume of hyperbolic orthoscheme tetrahedron as function of three dihedral angles.
On the other hand, Stackel says the following:
A simple derivation of the formula for the element $dP$ of a pyramid was given by Lobachevsky in year 1830 in the treatise "on the principles of geometry". He writes: $$dP = \frac{1}{2}d\psi(r\mathbb{cos}\varphi-h)$$
In fact, in non-euclidean geometry $r\mathbb{cos}\varphi$ is greater than $h$ or $x$. On the other hand, he does not explicitly mention the formula for volume $ABCabc$, which Gauss seems to have had as early as 1832 (see p.229 of this volume). It also reads as: the volume ABCabc is equal to half the product of the base $BC$ and the area of the infinitely small quadrilateral $ADda$.
In Vinberg's book mentioned before, he too does not mention Lobachevsky or Bolyai as the originators of the volume formula for hyperbolic wedge, and just remarks that Schlafli came close to it in 1853.
If Stackel and Vinberg did not confuse the historical facts, than I think the formula for volume of hyperbolic wedge is perhaps the only result in hyperbolic geometric that originated with Gauss; all his other unpublished results were covered and developed much more comprehensively by Bolyai and Lobachevsky. As such, it is an important fact.
(I know this post is very long, but I think this is the only place in Web where Gauss's work on hyperbolic volume is discussed comprehensively, so I had to collect all useful information here.)
