Modular Tensor Categories: Reasoning behind the axioms (Sorry for the length of the question, I'm trying to communicate what is bothering me as thoroughly as possible)
In the construction of modular tensor categories (MTC) from ground zero, we put structures one by one:


*

*Tensor Product Structure $\to$ Monoidal Categories

*Dual Objects $\to$ Rigid Monoidal Categories

*Representation of Braid Group $\to$ Rigid Braided Monoidal Categories

*Abelian and $k$-Linear Category

*Semisimplicty

*Finiteness conditions $\to$ Locally Finiteness and Finite Number of Isomorphism Class of Simple Objects

*$\mathrm{End}(\mathbf{1})=k$: The unit object is simple.

*$X\simeq X^{**}$ $\to$ Pivotal Categories
Demanding the above we get a Ribbon Fusion Category. If we look back at these structures there is a very good and mathematically natural reason why we demand them. The most non-trivial reasons are probably related to $\mathrm{End}(\mathbf{1})=k$ and finiteness conditions. But even then there are good reasons, for example $\mathrm{End}(\mathbf{1})=k$ guarantees not only that unit is simple, but also that quantum traces are a $k$-number. The finiteness conditions on the other hand gives us access to Jordan-Holder and Krull-Schmidt theorems, and isomorphisms like $(X\otimes Y)^*\simeq Y^*\otimes X^*$. So every last demand is quite reasonable and natural for a mathematician to study!
Then comes the last structure of an MTC:

The following structures on a ribbon fusion category are equivalent and this last structure defines a Modular Tensor Category
  
  
*
  
*(Turaev) The matrix $\tilde{s}$ with components $\tilde{s}_{ij}=\mathrm{tr}(\sigma_{{X_i}X_j}\circ \sigma_{{X_j}X_i})$
  is invertible (trace is quantum trace, $\sigma$ is braiding
  isomorphism and $X_i$ are simple)
  
*(Bruguières, Müger) Multiples of unit $\mathbf{1}$ are the only transparent objects of the category. (An object $X$ is called
  transparent if $\sigma_{XY}\circ\sigma_{YX}=\mathrm{id}_{Y\otimes X}$
  for all objects $Y$.)
  
*Our premodular category $\mathscr{C}$ is factorizable, i.e. the functor $\mathscr{C}\boxtimes \mathscr{C}^\text{rev}\to
 \mathcal{Z}(\mathscr{C})$ is an equivalence of categories.
  

I've been desperately searching for a good reason on why this last demand is also mathematically natural to ask. If I may be so bold, seems a bit random to me! The best I could came up with is that

An MTC is the polar opposite to a symmetric ribbon fusion category. In other words in a symmetric category every object is transparent, so it is, in a way, maximally transparent. While an MTC is minimally transparent (maximally opaque maybe!).

But this still doesn't satisfy my irk! So what? What about the categories in between? Then there are indirect reasons why one should demand this structure:

By demanding this last structure a lot of `nice' things happen.
  
  
*
  
*We have a projective representation of $\mathrm{SL}(2,\mathbb{Z})$ in our MTC: Namely matrices
  $s=\tilde{s}/\sqrt{D}$ with $D$ being the global dimension, $t$ with
  $t_{ij}=\theta_i\delta_{ij}$ and $c$ (charge conjugation)
  $c_{ij}=\delta_{ij^*}$ such that $$(st)^3=\zeta^3s^2, \quad s^2=c,
\quad ct=tc, \quad c^2=1$$
  
*Modular Functors and their $\mathscr{C}$-extensions and they are very closely related to MTCs.
  
*Verlinde Formula: The data $s,t,\zeta$ determines an MTC uniquely.
  
*Vafa Theorem and it's generalizations: The scalars $\theta_i$ and $\zeta$ are roots of unity.
  
  
  and even more nice things...

But still... Unless there is some good reason why we demand this last conditions naturally, all of this seems like a happy coincidence. So basically my question is

Is there a deep fundamental mathematical (or physical, physical is also fine) reason behind this last structure, which makes this condition a priori?

 A: Interesting question ! As far as I know, there are at least two secretly equivalent answers. 
You somehow already gave the first one: a modular tensor category is the same as a modular functor (though the precise statement is quite subtle, see the beautiful introduction to this paper : https://arxiv.org/abs/1509.06811). A modular functor is, roughly, a collection of compatible (projective) representations of mapping class groups, so in particular you need a representation of $SL_2(\mathbb{Z})$. It turns out this is also sufficient, in the sense that a premodular tensor category gives representations of mapping class groups in genus 0, and modularity is exactly what you need to extends this to higher genus.
A somehow more conceptual reason is related to the fact that those theories have an anomaly, i.e. you get only projective representations of MCG's, and invariant of 3-fold with some extra structure (I learned this point of view from Walker and Freed-Teleman). The origin of the story is the 3d Chern-Simons TFT introduced by Witten using Feynman integrals. It turns out that there is a 4d TFT around, a very simple topological version of Yang-Mills theory. This theory is simple because it involves integrating 4-forms on 4-folds, and those forms are actually exact. So if Feynman integrals really were integrals, by Stokes theorem this would be trivial on closed manifold, and for a 4-fold W with boundary we could define
$$Z_{CS}(\partial W):=Z_{YM}(W).$$
SInce any oriented 3-fold bound a 4-fold this would indeed be enough to define your theory.
It turns out $Z_{YM}$ is not trivial, but it is close to be: it is an invertible theory. It means in particular that it attaches 1-dimensional vector spaces to 3-fold, and that every 4-fold $W$ gives an isomorphism
$$Z_{YM}(\partial W) \longrightarrow Z_{YM}(\emptyset)=\mathbb{C}$$
which depends on the choice of $W$ only up to bordism. Hence you get a number defined up to this choice, and the bordism class of $W$ is precisely the extra structure occurring in this anomaly I was talking about.
How does it relates to your question ? Well, it is expected that every premodular category gives rise to a 4-dimensional TFT. Roughly speaking, to know whether this theory is invertible, it's enough to check what you get for the 2-sphere, which is a category, and it turns out this category is equivalent to the Müger center, i.e. the category of transparent objects. At the categorical level, invertibility means being equivalent to the category of vector spaces. 
Therefore, the modularity condition is exactly what you need to go from a 4d TFT to a 3d TFT with anomaly. 
