I think Carl Pomerance gets very close to giving a complete answer to the first part of you question, in the survey mentioned by Carlo Beenakker in the comments.

Perhaps it is not clear why the number field sieve is a good factoring
algorithm. A key quantity in a factorization method such as the
quadratic sieve or the number field sieve is what I was calling *"$X$"*
earlier. It is an estimate for the size of the auxiliary numbers that
we are hoping to combine into a square. Knowing $X$ gives you the
complexity; it is about $\exp(\sqrt{2 \log X \log \log X})$. In the
quadratic sieve we have X about $n^{1/2+\epsilon}$. But in the number
field sieve, we may choose the polynomial $f(x)$ and the integer $m$
in such a way that $(a-mb)N(a-\alpha b)$ (the numbers that we hope to
find smooth) is bounded by a value of $X$ of the form $\exp (c' (\log
> n)^{2/3})(\log \log n)^{1/3}))$. Thus the number of digits of the
auxiliary numbers that we sieve over for smooth values is about the
$2/3$ *power* of the number of digits of $n$, as opposed to the
quadratic sieve where the auxiliary numbers have more than half the
number of digits of $n$. That is why the number field sieve is
asymptotically so fast in comparison.

Pages from 7 to 10 of the survey are particularly relevant, although the whole paper is defenitely worth reading.

For a more technical and concrete exposition, you might be interested in:

As for the second part, question of the type "why can't we prove/improve __" are usually hard or impossible to answer, except in cases where there is a well know obstruction.

In particular, we can't rule out that the number field sieve is the best possible. Take for example this fragment from Anirban Pathak's "Elements of Quantum Computation and Quantum Communication":

[...] This indicates that factorization is a computational problem which
does not belong to $P$ and belongs to $NP$. This suggests that $P \subsetneq NP$. We reach this conclusion by considering the number
field sieve algorithm as the best algorithm. However, we cannot
exclude the possibility that tomorrow someone may invent an efficient
classical algorithm for factorization.