Why doesn't this construction of the tangent space work for non-Riemannian metric manifolds? In the 1957 paper, On the differentiability of isometries, Richard S. Palais gives a way to construct the tangent spaces of a Riemannian manifold using only its metric space structure (Theorem, p.1).
Specifically, given a Riemannian manifold $M$ and a point $p \in M$, the tangent space $T_p M$ equals the group of germs of pointed local similitudes $\mathbb{R} \to M$, $0 \mapsto p$. (Definitions below.)

Question: What are the metric space properties of Riemannian manifolds which allow this construction to work for Riemannian manifolds but not for arbitrary metric manifolds?

This construction can't work for arbitrary metric manifolds because any second-countable, Hausdorff, locally Euclidean space is metrizable, and yet there exist such spaces which are not homeomorphic to any smooth manifold, and in particular have no good definition of tangent spaces. If this construction worked generally, these manifolds would have tangent spaces; contradiction.
Definitions: Given metric spaces $(M_1,d_1)$ and $(M_2, d_2)$, a similitude is a bijective map $f:M_1 \to M_2$ such that, for a particular $r > 0$, for all $x,y \in M_1$, $d_2(f(x),f(y))= r \cdot d_1(x,y)$. An isometry is just the special case when $r = 1$.
A local similitude is the obvious analog of a local isometry (see Burago, Burago, Ivanov, Metric Geometry, Definition 3.4.1., p.78). Namely, a map $f: M_1 \to M_2$ is called a (pointed) local similitude at $x \in M_1$ (to $p \in M_2$) if $x \in M_1$ has an open neighborhood $U_x \subseteq M_1$ such that the (restriction of) $f$ is a similitude $U_x \to V_p$, where $V_p$ is some open neighborhood of $p$.
(Pointed here is solely meant to denote here that the point $p \in M_2$ under consideration is fixed in advance, as opposed to a "non-pointed" local similitude at $x \in M_1$, which could map onto any open set in $M_2$, not necessarily a neighborhood of the specified point $p \in M_2$. The terminology admittedly probably makes more sense when $f: M_1 \to M_1$ is a self-map, and $p = x$.)
Germ here is meant to denote the usual equivalence relation, where two functions belong to the same germ if and only if they agree on some open neighborhood of the point in question.
Note: Needless to say, Richard S. Palais uses different terminology. I am fairly confident, but not 100% confident, that my characterization is accurate/equivalent. (See here.) It is at the very least superficially similar to the definition of tangent spaces for smooth manifolds (germs of smooth maps $\mathbb{R} \to M$, $0 \mapsto p$ under an equivalence relation of smooth jets, see here). Tangent vectors are used to formalize notions of direction (see), and I had already been lead to germs of pointed local similitudes when trying to formalize the intuition of "direction" from Euclidean space (albeit ones $M \to M, p \mapsto p$ rather than $\mathbb{R} \to M, 0 \mapsto p$), see here (although it needs to be revised further).
Even if my attempted characterization of his result is not equivalent, I am still interested in the construction Richard S. Palais mentions for the tangent space of a Riemannian manifold, and would still like to know which metric space properties (e.g. strictly intrinsic metric? length space? existence of geodesics? etc.) allow the construction to work for Riemannian manifolds but not for arbitrary metric space manifolds. It would be interesting in particular to observe whether those metric space properties also hold for, e.g., Finsler manifolds.
Note: This website is for "research-level" mathematics. However, this question is with reference to a fairly old paper (1957), so I am not sure if it is on-topic. If not, please tell me. Note that there are at least two other questions (here and here) on MathOverflow which mention/discuss the paper.
This question seems at least tangentially related.
 A: Pretty much the papers to read are
Nikolaev, I.G., Smoothness of the metric of spaces with two-sided bounded Aleksandrov curvature, Sib. Math. J. 24, 247-263 (1983); translation from Sib. Mat. Zh. 24, No.2(138), 114-132 (1983). ZBL0547.53011.  
Nikolaev, I.G., A metric characterization of Riemannian spaces, Sib. Adv. Math. 9, No.4, 1-58 (1999). ZBL0956.53027. See also MR2142160
The point is that the Riemannian distance function $d$ (when we do not even know in advance that the underlying space topological space $X$ admits structure of a smooth manifold) satisfies a number of properties (which are discussed in the "Metric Geometry" book that you are reading), such as:


*

*$(X,d)$ is a path-metric and (locally) a geodesic metric space. 

*$(X,d)$ has (locally) curvature bounded above and below (where the upper and lower bounds are in the sense of the comparison geometry).

*$(X,d)$ has (locally) extendible geodesics. 

*... (local compactness, etc.) 
Among other things, the (local) curvature bounds guarantee that geodesics between two points are (locally) unique. The point of the two examples I suggested (in my comment) is to see what goes wrong if one drops these assumptions, in particular, what happens to the "tangent space" that you defined (e.g. it may fail to have structure of a vector space; its dimension might become larger than the topological dimension of $X$, e.g. infinite). A cone is just a regular cone of revolution in $E^3$, obtained by rotating a ray $R$ along another ray $A$ (the axis of rotation), such that $R$ and $A$ share the initial point (the apex of the cone); you equip this cone  with the path-metric induced from $E^3$. In both examples, the underlying space is a topological manifold, but something goes wrong with its "tangent space" that you defined: In the first example it is infinite-dimensional, in the second example it is not a vector space. 
What Nikolaev proved is quite remarkable: The above properties imply that $(X,d)$ is isometric to a manifold equipped with a Riemannian distance function, in particular, the tangent space (at every point) indeed has natural structure of a vector space of the expected dimension. (The difference between the two papers is in the degree of smoothness of the Riemannian metric: The first paper gives a $C^{1,\alpha}$-smooth metric, while the second paper gives a $C^{\infty}$ metric, under a stronger hypothesis.) This result was conjectured by A.D.Alexandrov in the 1940s. Nikolaev's papers are a  culmination of a long chain of results, including, for instance, the one by Berestovsky who proved the existence of a continuous metric, etc. 
A: This is community wiki; this also does not answer the question. I just want to post some quotes from Burago, Burago, Ivanov which seem possibly relevant here. If upon further reflection I can think of more to say about them, or realize they are irrelevant to the question at hand, I will modify or delete this CW "answer" as appropriate. Cluttering up the question further seemed inadvisable.
pp. 140-143
(discussing/referring to arbitrary Finsler manifolds)

As usual, the length structure $L$ gives rise to an intrinsic metric $d$. It is easy to show that the length of a curve induced by $d$ coincides with $L$ for all smooth curves (Exercise 5.1.1). However, whereas the existence of a shortest path for $d$ connecting any two points follows from Theorem 2.5.23, its smoothness is less obvious. Moreover, a shortest path for $d$ can fail to be smooth unless a certain additional assumption of strict convexity is imposed on $\lambda_p$.
Such a pathology (nonsmoothness and nonuniqueness of shortest paths, even in an arbitrary small neighborhood of a point) is due to the fact that balls of the norm (which are "diamonds") are not strictly convex [emphasis mine]. An interested reader can prove that for strictly convex norms all shortest paths are smooth, and a shortest path connecting two sufficiently close points is unique. 
Hint: Use an analog of Lemma 5.1.13 with an appropriate two-dimensional normed space instead of Euclidean space.

...

Remark 5.1.5. Isometric Riemannian regions are obviously isometric as length spaces. The converse is also true: if Riemannian regions $\Omega$ and $\Omega'$ are isometric as length spaces, then they are isometric in the sense of Definition 5.1.4 (that is, there exists a smooth isometry that respects their Riemannian structures). Moreover, every isometry map from $\Omega$ to $\Omega'$ is smooth, and (as a consequence) can be taken as $\varphi$ in Definition 5.1.4.
We leave this fact as an (not so obvious!) exercise. The easiest proof we know is based on the results of Section 5.2 (namely, smoothness of shortest paths and properties of exponential maps).
Smoothness of isometries allows us to give a metric definition of a Riemannian manifold (for we were so far looking only at regions with Riemannian metrics):
Definition 5.1.6. A Riemannian manifold is a length space such that every point has a neighborhood isometric to a region with Riemannian metric.
Remark 5.1.7. This definition is not standard. In most textbooks Riemannian manifolds are defined as smooth manifolds equipped with Riemannian structures. The definitions are equivalent: this easily follows from Remark 5.1.5 (in particular, a length space that is locally isometric to a Riemannian region naturally carries a structure of a smooth manifold). If you are familiar with smooth manifolds, we recommend you prove this equivalence as an exercise.

What I take away from all of this is (at least) the following:


*

*The fact that Riemannian manifolds are length spaces is crucial to Palais's construction.

*The fact that the metric on Riemannian manifolds is not only intrinsic but even strictly intrinsic is also crucial to Palais's construction. (Also called a geodesic space? (cf.) BBI uses different terminology.) Anyway, this also implies (I think) that Riemannian manifolds are convex metric spaces (not to be confused with geodesic convexity).

*The fact that the metric balls of Riemannian manifolds are strictly convex, while those of arbitrary Finslerian manifolds are not, may also be crucial.

*The fact that all isometries (and thus presumably also all local similitudes) of Riemannian manifolds are smooth is crucial.

*The fact that all geodesics (or at least of shortest paths) are smooth is crucial. Namely, we can replace as representatives of (many) germs in the standard construction of the tangent space with geodesics (of possibly varying speeds). (Although the fact that every germ has as a representative some geodesic of some speed is surprising to me, but anyway if this is true I believe that it is almost certainly related to the fact that geodesics are smooth.)

*Riemannian manifolds have this nice length functional $L(\gamma, a , b ) = \int_a^b \lambda_{\gamma(t)}(\gamma'(t))dt$, where $\lambda_p(v) = \sqrt{Q_p(v,v)}$ for some positive definite (non-degenerate?) quadratic form $Q_p$ -- although this is ridiculously circular, because this already assumes the existence of a smooth manifold structure, i.e. the existence of tangent spaces at every point and thus tangent vectors $v$ to populate those tangent spaces and thus the ability to define length functionals which take tangent vectors as arguments. 
So for me, the primary concerns/confusions are the following:


*

*These concerns help to show how Palais's construction coincides with the standard construction for smooth manifolds. They do not show how Palais's construction will be degenerate for metric spaces which are not Riemannian manifolds. In fact, the smoothness of geodesics/shortest paths, the strictly convex balls, and the strictly intrinsic metrics all suggest that this construction might work for arbitrary Finslerian manifolds as well.

*What about Finslerian manifolds where the norms have balls which are strictly convex, but which aren't induced by an inner product. Is there a theorem which says that the balls of a normed space are strictly convex if and only if the norm is induced by an inner product? (I know that the norm is isotropic if and only if it is induced by an inner product, but I don't see how that is related to strict convexity of its unit balls.) This suggests there is no relationship, since balls of $p$-norms for $1<p<\infty$ (a large range) are strictly convex.)

*I vaguely recall there being a relationship between sub-additivity and lower semi-continuity of the length functional (this being mentioned in Burago, Burago, Ivanov), as well as a relationship with convexity of the unit balls and all of this being related to (semi- or quasi- or asymmetric) norms somehow (mentioned elsewhere). The answers might be found in pp. 200-208 of Volume 2 of Spivak (the Addendum about Finsler geometry). Maybe there is a relationship with being CAT(0)?


(Wikipedia 1 2)

Furthermore, [a convex set] $C$ is strictly convex [emphasis mine] if every point on the line segment [geodesic?] connecting $x$ and $y$ other than the endpoints is inside the interior of $C$.

-

The definition of a convex set and a convex hull extends naturally to geometries which are not Euclidean by defining a geodesically convex  set [link added by me] to be one that contains the geodesics joining any two points in the set.

