Here is an example of a total injective function $f:\mathbb{N}\to\mathbb{N}$, which is computable in polynomial time, but whose inverse function is not even computable, let alone polynomial-time computable. The idea is simply that we treat the input $n$ as a existential witness for some undecidable property of a smaller number $k$, such as the halting problem. Since these witnesses can and indeed must become very large, the inverse function, as Darij expected, will have large blow-up in size.
Let $f(n)=2k$ if $n$ is the code, in some highly canonical way, of a sequence of Turing machine configurations of the complete computation of program $k$ on input $0$ showing it to halt. Otherwise, if $n$ is not such a code, we let $f(n)=2n+1$.
This function is easily computable, in low-degree polynomial time, since we can check if $n$ is such a code easily and then give the corresponding output. The function is total and injective, since we give even output in the first case, in which case $n$ is determined by $k$ since we are using canonical codes, and $k$ is determined by $2k$, and we give odd output in the second case, in which case $n$ is again determined by $2n+1$.
But meanwhile, the inverse function is not computable at all, let alone in polynomial time, since we cannot tell whether $2k$ is in the range of the function unless we know whether $k$ halts on input $0$, which would solve the halting problem.
One can use essentially the same idea to give a direct non-polynomial time example. Namely, let
$g(1^{2^k})= 1^{2k}$, meaning sequences of $1$s, and $g(n)=1^{2n+1}$, if $n$ is not a sequence of $1$s of length a power of $2$. We can compute $g$ in polynomial time, but the inverse function takes exponential time.