The word "true" can, however, be defined mathematically. Truth is a property of sentences. If you have defined a formal language $L$, such as the first-order language of arithmetic, then you can define a sentence $S$ in $L$ to be true if and only if $S$ holds of the natural numbers. So for example the sentence $\exists x: x > 0$ is true because there does indeed exist a natural number greater than 0. Here it is important to note that true is not the same as provable. The formal sentence corresponding to the twin prime conjecture (which I won't bother writing out here) is true if and only if there are infinitely many twin primes, and it doesn't matter that we have no idea how to prove or disprove the conjecture.
Now, how can we have true but unprovable statements? And if we had one how would we know? Joel David Hamkins explained this well, but in brief, "unprovable" is always with respect to some set of axioms. Therefore it is possible for some statement to be true but unprovable from some particular set of axioms $A$. In order to know that it's true, of course, we still have to prove it, but that will be a proof from some other set of axioms besides $A$.