Return to Answer

Second, if $\nabla$ and $\tilde\nabla$ are two linear connections on $TM$, their difference is well-defined and is a section of $TM\otimes T^*M\otimes T^*M$. Specifically $\tilde\nabla - \nabla:TM\times TM\to TM$ has the property that, on vector fields $X$ and $Y$, we have $$ \bigl(\tilde\nabla - \nabla\bigr)(X,Y) = \tilde\nabla_XY-\nabla_XY. $$

Lemma 2: Two linear connections, $\nabla$ and $\tilde\nabla$ have the same geodesics (i.e., each curve $\gamma$ is a geodesic for one if and only if it is a geodesic for the other) if and only if $\tilde\nabla - \nabla$ is a section of the subbundle $TM\otimes\Lambda^2(T^*M)\subset TM\otimes T^*M\otimes T^*M$.

Lemma 3: If $g$ is a nondegenerate (pseudo-)Riemannian metric, and $\nabla$ and $\tilde\nabla$ are linear connections on $TM$ that satisfy $\nabla g = \tilde\nabla g = 0$, then they have the same geodesics if and only if the expression $$ \phi(X,Y,Z) = g\bigl( X,(\tilde\nabla{-}\nabla)(Y,Z)\bigr) $$ is skew-symmetric in $X$, $Y$, and $Z$.

Proof: $\nabla g = \tilde\nabla g = 0$ implies $\phi(X,Y,Z)+\phi(Z,Y,X)=0$, while they have the same geodesics if and only if $\phi(X,Y,Z)+\phi(X,Z,Y)=0$.

Second, if $\nabla$ and $\nabla^*$ are two linear connections on $TM$, their difference is well-defined and is a section of $TM\otimes T^*M\otimes T^*M$. Specifically $\nabla^* - \nabla:TM\times TM\to TM$ has the property that, on vector fields $X$ and $Y$, we have $$ \left({\nabla^*} - \nabla\right)(X,Y) = {\nabla^*}_XY-\nabla_XY. $$

Lemma 2: Two linear connections, $\nabla$ and $\nabla^*$ have the same geodesics (i.e., each curve $\gamma$ is a geodesic for one if and only if it is a geodesic for the other) if and only if $\tilde\nabla - \nabla$ is a section of the subbundle $TM\otimes\Lambda^2(T^*M)\subset TM\otimes T^*M\otimes T^*M$.

Lemma 3: If $g$ is a nondegenerate (pseudo-)Riemannian metric, and $\nabla$ and $\nabla^*$ are linear connections on $TM$ that satisfy $\nabla g = \nabla^*g = 0$, then they have the same geodesics if and only if the expression $$ \phi(X,Y,Z) = g\bigl( X,(\nabla^*{-}\nabla)(Y,Z)\bigr) $$ is skew-symmetric in $X$, $Y$, and $Z$.

Proof: $\nabla g = \nabla^* g = 0$ implies $\phi(X,Y,Z)+\phi(Z,Y,X)=0$, while they have the same geodesics if and only if $\phi(X,Y,Z)+\phi(X,Z,Y)=0$.

Remark on connections with the same geodesics: I realize that I didn't respond to the OP's confusion about connections with the same geodesics vs. compatible with a metric $g$ but with torsion. (I did respond in a comment that turned out to be wrong, so I deleted it. Hopefully, this will be better.)

First, about torsion (of a connection on TM). The torsion $T^\nabla$ of a (linear) connection on $TM$ is a section of the bundle $TM\otimes\Lambda^2(T^*M)$. Here is an (augmented) Fundamental Lemma of (pseudo-)Riemannian geometry:

Lemma 1: If $g$ is a (nondegenerate) pseudo-Riemannian metric on $M$ and $\tau$ is a section of $TM\otimes\Lambda^2(T^*M)$, then there is a unique linear connection $\nabla$ on $TM$ such that $\nabla g = 0$ and $T^\nabla = \tau$.

(The usual FLRG is the special case $\tau=0$.) Note that this $\nabla$ depends algebraically on $\tau$ and the $1$-jet of $g$. The proof of Lemma 1 is the usual linear algebra.

Second, if $\nabla$ and $\tilde\nabla$ are two linear connections on $TM$, their difference is well-defined and is a section of $TM\otimes T^*M\otimes T^*M$. Specifically $\tilde\nabla - \nabla:TM\times TM\to TM$ has the property that, on vector fields $X$ and $Y$, we have $$ \bigl(\tilde\nabla - \nabla\bigr)(X,Y) = \tilde\nabla_XY-\nabla_XY. $$

Lemma 2: Two linear connections, $\nabla$ and $\tilde\nabla$ have the same geodesics (i.e., each curve $\gamma$ is a geodesic for one if and only if it is a geodesic for the other) if and only if $\tilde\nabla - \nabla$ is a section of the subbundle $TM\otimes\Lambda^2(T^*M)\subset TM\otimes T^*M\otimes T^*M$.

Proof: In local coordinates $x = (x^i)$, let $\Gamma^i_{jk}$ (respectively, $\tilde\Gamma^i_{jk}$) be the coefficients of $\nabla$0 (respectively, $\tilde\nabla$). Then $$ \tilde\nabla-\nabla = (\tilde\Gamma^i_{jk}-\Gamma^i_{jk})\ \partial_i\otimes \mathrm{d}x^j\otimes\mathrm{d}x^k. $$ Meanwhile, a curve $\gamma$ in the $x$-coordinates is a $\nabla$-geodesic (respectively, a $\tilde\nabla$-geodesic) iff $$ \ddot x^i + \Gamma^i_{jk}(x)\,\dot x^j\dot x^k = 0\qquad (\text{respectively},\ \ddot x^i + \tilde\Gamma^i_{jk}(x)\,\dot x^j\dot x^k = 0). $$ These are the same equations iff $(\tilde\Gamma^i_{jk}(x)-\Gamma^i_{jk}(x))\,y^jy^k\equiv0$ for all $y^i$, i.e., iff $$ {\tilde\nabla}-\nabla = \tfrac12({\tilde\Gamma}^i_{jk}-\Gamma^i_{jk})\ \partial_i\otimes \mathrm{d}x^j\wedge\mathrm{d}x^k.\quad \square $$

Finally, we examine when two $g$-compatible connections have the same geodesics:

Lemma 3: If $g$ is a nondegenerate (pseudo-)Riemannian metric, and $\nabla$ and $\tilde\nabla$ are linear connections on $TM$ that satisfy $\nabla g = \tilde\nabla g = 0$, then they have the same geodesics if and only if the expression $$ \phi(X,Y,Z) = g\bigl( X,(\tilde\nabla{-}\nabla)(Y,Z)\bigr) $$ is skew-symmetric in $X$, $Y$, and $Z$.

Proof: $\nabla g = \tilde\nabla g = 0$ implies $\phi(X,Y,Z)+\phi(Z,Y,X)=0$, while they have the same geodesics if and only if $\phi(X,Y,Z)+\phi(X,Z,Y)=0$.

Corollary: If $g$ is a nondegenerate (pseudo-)Riemannian metric, then the space of linear connections $\nabla$ on $TM$ that satisfy $\nabla g = 0$ and have the same geodesics as $\nabla^g$, the Levi-Civita connection of $g$, is a vector space naturally isomorphic to $\Omega^3(M)$, the space of $3$-forms on $M$.

I think that the literal answer is that the Levi-Civita connection of $g$ is trying to describe the metric $g$ and nothing else. It is the only connection-assignment that is uniquely defined by the metric and its first derivatives and nothing else, in the sense that, if you have a diffeomorphism-equivariant assignment $g\to C(g)$ where $C(g)$ is a connection that depends only on $g$ and its first derivatives, then $C(g)$ is the Levi-Civita connection.

Note that the restriction to first derivatives is necessary. For example, there is a unique connection on $TM$ that is compatible with $g$ and satisfies $$ \nabla_XY -\nabla_YX - [X,Y] = \mathrm{d}S(X)\,Y - \mathrm{d}S(Y)\,X, $$ where $S= S(g)$ is the scalar curvature of $g$. However, this canonical connection depends on three derivatives of $g$.

Meanwhile, connections with torsion can arise naturally from other structures: For example, on a Lie group, there is a unique connection for which the left-invariant vector fields are parallel and a unique connection for which the right-invariant vector fields are parallel. When the identity component of the group is nonabelian, these are distinct connections with nonvanishing torsion, while their average is a canonical connection that is torsion-free. (This latter connection need not be metric compatible, of course.) A more well-known example is the unique connection associated to an Hermitian metric on a complex manifold that is compatible with both the metric and the complex structure and whose torsion is of type (0,2).

It's not unreasonable to ask whether imposing the torsion-free condition, just because you can, right out of the gate is too restrictive. Einstein tried for years to devise a 'unified field theory' that would geometrize all of the known forces of nature by considering connections compatible with the metric (i.e., the gravitational field) that had with torsion. There is a book containing the correspondence between Einstein and Élie Cartan (Letters on absolute parallelism) in which Einstein would propose a set of field equations that would constrain the torsion so that they describe the other known forces (just as the Einstein equations constrain the gravitational field) and Cartan would analyse them to determine whether they had the necessary 'flexibility' to describe the known phenomena without being so 'flexible' that they couldn't make predictions. It's very interesting reading.

This tradition of seeking a physical interpretation of torsion has continued, off and on, since then, with several attempts to generalize Einstein's theory of relativity. Some of these are described in Misner, Thorne, and Wheeler, and references are given to others. In fact, quite recently, Thibault Damour (IHÉS), famous for his work on black holes, and a collaborator have been working on a gravitational theory-with-torsion, which they call 'torsion bigravity'. (See arXiv:1906.11859 [gr-qc] and arXiv:2007.08606 [gr-qc].)

I guess the point is that 'why impose torsion-free?' is actually a very reasonable question to ask, and, indeed, it has been asked many times. One answer is that, if you are only trying to understand the geometry of a metric, you might as well go with the most natural connection, and the Levi-Civita connection is the best one of those in many senses. The other answer is that, if you have some geometric or physical phenomenon that can be captured by a metric and another tensor that can be interpreted as (part of) the torsion of the connection, then, sure, go ahead and incorporate that information into the connection and see where it leads you.

I think that the literal answer is that the Levi-Civita connection of $g$ is trying to describe the metric $g$ and nothing else. It is the only connection-assignment that is uniquely defined by the metric and its first derivatives and nothing else, in the sense that, if you have a diffeomorphism-equivariant assignment $g\to C(g)$ where $C(g)$ is a connection that depends only on $g$ and its first derivatives, then $C(g)$ is the Levi-Civita connection.

Note that the restriction to first derivatives is necessary. For example, there is a unique connection on $TM$ that is compatible with $g$ and satisfies $$ \nabla_XY -\nabla_YX - [X,Y] = \mathrm{d}S(X)\,Y - \mathrm{d}S(Y)\,X, $$ where $S= S(g)$ is the scalar curvature of $g$. However, this canonical connection depends on three derivatives of $g$.

Meanwhile, connections with torsion can arise naturally from other structures: For example, on a Lie group, there is a unique connection for which the left-invariant vector fields are parallel and a unique connection for which the right-invariant vector fields are parallel. When the identity component of the group is nonabelian, these are distinct connections with nonvanishing torsion, while their average is a canonical connection that is torsion-free. (This latter connection need not be metric compatible, of course.) A more well-known example is the unique connection associated to an Hermitian metric on a complex manifold that is compatible with both the metric and the complex structure and whose torsion is of type (0,2).

It's not unreasonable to ask whether imposing the torsion-free condition, just because you can, right out of the gate is too restrictive. Einstein tried for years to devise a 'unified field theory' that would geometrize all of the known forces of nature by considering connections compatible with the metric (i.e., the gravitational field) that had torsion. There is a book containing the correspondence between Einstein and Élie Cartan (Letters on absolute parallelism) in which Einstein would propose a set of field equations that would constrain the torsion so that they describe the other known forces (just as the Einstein equations constrain the gravitational field) and Cartan would analyze them to determine whether they had the necessary 'flexibility' to describe the known phenomena without being so 'flexible' that they couldn't make predictions. It's very interesting reading.

This tradition of seeking a physical interpretation of torsion has continued, off and on, since then, with several attempts to generalize Einstein's theory of gravity (aka, 'general relativity'). Some of these are described in Misner, Thorne, and Wheeler, and references are given to others. In fact, quite recently, Thibault Damour (IHÉS), famous for his work on black holes, and a collaborator have been working on a gravitational theory-with-torsion, which they call 'torsion bigravity'. (See arXiv:1906.11859 [gr-qc] and arXiv:2007.08606 [gr-qc].) [To be frank, though, I'm not aware that any of these alternative theories have made any predictions that disagree with GR that have been verified by experiment. I think we all would have heard about that.]

I guess the point is that 'why impose torsion-free?' is actually a very reasonable question to ask, and, indeed, it has been asked many times. One answer is that, if you are only trying to understand the geometry of a metric, you might as well go with the most natural connection, and the Levi-Civita connection is the best one of those in many senses. Another answer is that, if you have some geometric or physical phenomenon that can be captured by a metric and another tensor that can be interpreted as (part of) the torsion of the connection, then, sure, go ahead and incorporate that information into the connection and see where it leads you.