If your topological field theory is at least once-extended, by which I mean it assigns values to $(n+1)$-manifolds, $n$-manifolds, and also to $(n-1)$-manifolds, than this cannot happen.
More precisely we have the following result:
Theorem: Suppose that $ Z: Bord_{n+1} \to C$ is an $(n+1)$-dimensional once-extended (oriented) topological field theory valued in the symmetric monoidal 2-category $C$. Then the value $Z(T^n)$ assigned to the $n$-torus is invertible if and only if the whole field theory is invertible (assigns invertible values to all manifolds in all dimnesions).
So in particular suppose $C$ is any 2-category that deloops the category of vector spaces (like linear categories, functors, transformations or algebras, bimodules, maps, etc). Then if the vector space assigned to the n-torus is 1-dimensional (hence invertible) than every $n$-manifold is assigned an invertible (i.e. one-dimensional) vector space. This also holds if you replace vector spaces with super vector spaces.
A version of this theorem also holds for bordism equipped with arbitrary tangential structures. You can read about it here:1511.01772.
The main ingredients in the proof are dimensional reduction and surgery. Dimensional reduction lets you relate theories of different dimensions and so you can attack the problem inductively. This already lets you prove that many bordisms take invertible values. Then next idea is that handle decompositions use handles which have at most codimension two corners. This means that if you are at least once-extended, then you can implement handle decompositions of (n+1)-manifolds, and hence surgery for n-manifolds, entirely in the bordism 2-category in categorical terms.
That is not the complete argument, but those are two of the main ideas.
I don't know what happens if your field theory is not extended. Even in dimension n=2 (n+1 =3) the OP's argument used that we assign categories to 1-manifolds. So if the theory is not extended, perhaps it is possible to assign an a 1-dimensional vector space to the 2-torus, but higher dimensional spaces to other surfaces? András Juhász has a classification of non-extended (2+1)-TQFTs, but it is a bit complicated. Perhaps it is possible to use his work to prove or disprove this in dimension (2+1)? I think that is an interesting question to explore.