Gauss's procedure leads to Bolyai's result on the volume of orthoscheme tetrahedron, as I'll show here. However, Gauss's result is a little bit more limited than Bolyai, since Gauss refers to an orthoscheme tetrahedron of which 4 of the 12 face angles of the tetrahedron are right (each face is an hyperbolic right triangle), while Bolyai refers to a slightly more general tetrahedron whose only 3 face angles ar right.
In order to help visualize the relations, I added here a pic of Gauss's note.
Preliminary discussion:
Relation to Schlafli formula
To see the connection between Schlafli formula and the first formula in Gauss's fragment, one needs to understand that Gauss thinks of the tetrahedron 1234 in such a way that the faces 124 and 134 are perpendicular and the edges 24 and 13 meet the intersection line 14 also at right angles. So if we draw a unit sphere centered at vertex 1, we generate a right spherical triangle 234 (this spherical triangle has right angle at vertex 4 - this angle is simply the dihedral angle 14) and side length 34 which is also equal to a right angle arc; hence, vertex 3 is the pole of arc 24 and therefore the angle at vertex 2 (which is dihedral angle 12) is also right. Therefore, both dihedral angles 14 and 12 don't contribute to the sum in Schlafli formula.
In addition, Gauss defines the tetrahedron in such a way that the angles at vertex 3 are constant (so that an "observer" in hyperbolic space which is located at vertex 3 sees the rest of the vertexes at constant lines of sight). Since the three face angles at vertex 3 correspond to the length of sides of a spherical triangle, and the dihedral angles at sides 31,32,34 correspond to the angles of this spherical triangle, one gets that constancy of face angles at vertex 3 implies constancy of the dihedral angles 31,32,34.
Therefore, only the dihedral angle of the side 24 changes. By repeating the same argument on a unit sphere (argument which we used on vertex 1) centered at vertex 4, we conclude that vertex 2 of the spherical triangle 123 is the pole of side 13 so two face angles at vertex 4 are right (face angles 142 and 342) and dihedral angle 24 is equal to face angle 341. All this leads directly to the first formula in Gauss's fragment (apart from a missing factor of $\frac {1}{2}$).
Derivation of the fundamental metrical relation for an orthoscheme
To derive the metrical relation $\alpha^2 \mathbb{cot}^2(341)-\beta^2\mathbb{tanh}^2(24) = 1$, we use the fact that all faces of the tetrahedron are hyperbolic right triangles. For the right hyperbolic triangle 134 we have:
$$\mathbb{cosh}(34) = \mathbb{cot}(431)\mathbb{cot}(341)$$
and for the right hyperbolic triangle 234 we have:
$$\mathbb{sinh}(34) = \mathbb{cot}(234)\mathbb{tanh}(24)$$
so therefore we have the equation:
$$\mathbb{cot}^2(431)\mathbb{cot}^2(341) - \mathbb{cot}^2(234)\mathbb{tanh}^2(24) = 1$$
Derivation of explicit volume formula from Gauss's formula:
For the sake of consistency, we denote the angles 431, 234, and 214 as $\alpha$, $\beta$ and $\gamma$, respectively. Now lets look at the link of vertex 3 of the tetrahedron: it is a spherical triangle whose two edges lengths are $\alpha$ and $\beta$ and one angle is $\gamma$ (it is the dihedral angle of edge 31 and it is also equal to $\gamma$). In addition the sides $\alpha$$, \beta$ of this spherical triangle are orthogonal to each other. Therefore, by a combination of the spherical sine theorem and the spherical pythagoras theorem, we get:
$$ \frac{{\mathbb{sin}(\mathbb{arccos}(\mathbb{cos}\alpha\cdot \mathbb{cos}\beta))}}{{\mathbb{sin} (\frac{\pi}{2}})} = \frac {{\mathbb{sin}\beta}}{{\mathbb{sin}\gamma}}$$, or:
$$(1) \mathbb{sin}\gamma = \frac {{\mathbb{sin}\beta}}{{\sqrt{{1 - (\mathbb{cos}\alpha \cdot \mathbb{cos}\beta)^2}}}}$$
Now, denote the length of side $24$ as $l_{24} = x$ and the angle $341$ as $\varphi$. Since $\varphi$ is related to $x$ by the equation $c^2_1 \mathbb{cot}^2\varphi - c^2_2\mathbb{tanh}^2x = 1$ (here $c_1 = \mathbb{cot}\alpha,c_2 = \mathbb{cot}\beta$), one can write:
$$\varphi = \mathbb{arccot}(\frac{\sqrt{1+c^2_2\mathbb{tanh}^2x}}{c_1})$$
Gauss's procedure for the calculation of the volume, which uses the relation $\partial \Delta = -\frac{1}{2}x d\varphi$, leads to the following integral:
$$\Delta = -\frac{1}{2}\int x d\varphi = -\frac{1}{2}\int x \frac{d\varphi}{dx}dx$$
so one can compute the derivative of $\varphi$ with respect to $x$ by an application of the chain rule:
$$-\frac{d\varphi}{dx} = \frac{1}{1+\frac{1+c^2_2\mathbb{tanh}^2x}{c^2_1}}\frac{c^2_2\mathbb{tanh}x\cdot \frac{1}{\mathbb{cosh}^2x}}{c_1\sqrt{1+c^2_2\mathbb{tanh}^2x}}$$
Now make a very long algebraic simplification:
$$ -\frac{d\varphi}{dx} = \frac{\mathbb{sinh}x (c^2_2/c_1)}{\mathbb{cosh}^2x(1+\frac{1+c^2_2\mathbb{tanh}^2x}{c^2_1})\mathbb{cosh}x\cdot c_2\sqrt{\frac{1}{c^2_2}+\mathbb{tanh}^2x}} = \frac{\mathbb{sinh}x (c_2/c_1)}{\mathbb{cosh}^2x(1+\frac{1+c^2_2\mathbb{tanh}^2x}{c^2_1})\sqrt{\frac{\mathbb{cosh}^2x}{c^2_2}+\mathbb{sinh}^2x}} = \frac{\mathbb{sinh}x (c_1/c_2)}{(c_1/c_2)^2(\mathbb{cosh}^2x(1+\frac{1+c^2_2\mathbb{tanh}^2x}{c^2_1}))\sqrt{\frac{\mathbb{cosh}^2x}{c^2_2}+\mathbb{sinh}^2x}}$$
The expression under the square root is $$\sqrt{\frac{\mathbb{cosh}^2x}{c^2_2}+(\mathbb{cosh}^2x-1)} = \sqrt{\mathbb{cosh}^2x(\frac{1}{c^2_2}+1)-1} = \sqrt{{\frac{{\mathbb{cosh}^2x}}{{\mathbb{cos}^2\beta}} - 1}}$$
while the expression in the left side of the denominator is equal to:
$$(c_1/c_2)^2\mathbb{cosh}^2x(1+\frac{1}{c^2_1})+\mathbb{sinh}^2x = \mathbb{cosh}^2x(1+\frac{c^2_1+1}{c^2_2}) -1 = \mathbb{cosh}^2x(1+\frac{1}{\mathbb{sin}^2\alpha \mathbb{cot}^2\beta})-1$$
Recalling that $\frac{c_1}{c_2} = \frac {\mathbb{tan}\beta}{\mathbb{tan} \alpha}$, the resulting expression for the integral is:
$$\Delta = \frac {{\mathbb{tan}\beta}}{{2 \mathbb{tan} \alpha}}\int_{0}^{c}\frac {{x \mathbb{sinh}(x) dx}}{{(\mathbb{cosh}^2x(1 + \frac {{1}}{{\mathbb{sin}^2\alpha \mathbb{cot}^2\beta}})-1)\sqrt{{\frac{{\mathbb{cosh}^2x}}{{\mathbb{cos}^2\beta}} - 1}} }}$$
Now, the left factor of the denominator $\mathbb{cosh}^2(x)(1 + \frac{{1}}{{\mathbb{sin}^2\alpha \ \mathbb{cot}^2\beta}})-1$, is exactly equal to $\mathbb{cosh}^2(x)\cdot \frac{{1}}{{\mathbb{cos}^2\gamma}}-1$, because subtitution of $\mathbb{cos}\gamma = \sqrt {1 - \frac {{\mathbb{sin}^2\beta}}{{1-(\mathbb{cos}\alpha\cdot \mathbb{cos}\beta)^2}}}$ (this substitution is true because of relation (1)) in this expression gives the previous one.
Concluding remarks:
- As can be seen from this presentation - Hyperbolic Volumes and Symmetry, the Bolyai's volume integral is written in my notation in this way (see Theorem 5, p. 12, at this presentation) :
$$Vol(T) = \frac {{\mathbb{tan}\beta}}{{2 \mathbb{tan} \alpha}}\int_{0}^{c}\frac {{x \mathbb{sinh}(x) dx}}{{(\frac {{\mathbb{cosh}^2(x)}}{{\mathbb{cos}^2\gamma}} - 1)\sqrt{{\frac{{\mathbb{cosh}^2x}}{{\mathbb{cos}^2\beta}} - 1}} }}$$
and in the case treated here Bolyai's integral coincides with the result of Gauss's procedure. Important Note: the differences in notation between the Bolyai integral in the presentation and Gauss's integral are just due the different symbols of the angles 431, 234, and 214 - $\alpha,\beta,\gamma$ in the presentation correspond to $\gamma, \alpha ,\beta$ in my notation.
However, for the case treated by Gauss, his formulas are absolutely correct. He should also be given credit for the identification of the calculation of the orthoscheme tetrahedron as the basis for volume formulas of general tetrahedrons (without right angles). In one of his letters, he refered to those calculations of volumes as "die jungle" - I guess he refered to the extremely complicated integrals that arise in the attempts to the decompose the general tetrahedron into orthoscemes (this problem was only solved very recently).
- Paul Stackel, the mathematician who edited Janos Bolyai's geometric works, had the following things to say about Bolyai's derivation of his integral formula:
It is most remarkable that the method that Gauss used for cubing the tetrahedron, is exactly the same as that of Johann. This is shown in a note from March 1832, from Gauss's estate, which is printed in the works (vol. VIII, p. 228); Gauss has exactly the same special tetrahedron (only 3142 instead $abc\delta$ means) and exactly the same decomposition by planes perpendicular to ab (31).
This quotation is taken from p. 113 of the book "Wolfgang und Johann Bolyai geometrische Untersuchungen" (here is a link: https://archive.org/details/wolfgangundjohan01stuoft/page/112/mode/2up), which was edited and translated to german by the Stackel.
- It's still necessary to understand how Gauss arrived at the formula $\partial \Delta = -\frac{{1}}{{2}}(24)d(341)$ (he missed the factor $\frac {{1}}{{2}}$ at the first attempt). In his commentary on Gauss's note, Stackel derives it in the following way:
The tetrahedron $1234$, whose volume is called $\Delta$, may now experience an infinitely small increase in volume $1 2 4 1' 2' 4' = \partial \Delta$, by lengthening the edge $31$ by the infinitely small amount $11' = d(13)$ and through $1'$ a perpendicular to $31'$ laying in the plane $1'2'4'$, which intersects the edges $32$ and $34$ in $2'$ and $4'$, respectively. The angles at the corner $3$, and thus also the sizes $\alpha$ and $\beta$, remain unchanged. The angle $(341)$ changes into the angle $$(34'1') = (3 4 1)+d(3 4 1)$$ namely, like the consideration of the quadrilateral $1 1' 4' 4$ with the infinitely small base line $1 1' = d(1 3)$ and right angles at $1$ and $1'$ recognized immediately: $$d(3 4 1) = \mathbb{sinh}(14)\cdot d(13)$$ The increase in volume $1 2 4 1' 2' 4'$ is bounded laterally by the triangles $1 2 4$ and $1' 2' 4'$, whose planes are both perpendicular to $1 1'$, and hence (see p.233 of this volume): $$\partial \Delta = -\frac{1}{2}d(13)\cdot(24)\mathbb{sinh}(14)$$ hence:$$\partial \Delta = -\frac {1}{2}(24)\cdot d(3 4 1) $$ and that is, apart from the missing factor $\frac{1}{2}$, Gauss's formula.
Since Stackel refers to Gauss's second fragment on volume determinations in non-euclidean geometry (p. 233 of he same volume), which was written in 1840 and was found next to Gauss's copy of one of Lobachevski's publications, I think understanding Gauss's second fragment may help understanding Gauss's reasoning.