Part of the reason for the confusion here is that the word "true" is sometimes used informally, and at other times it is used as a technical mathematical term.
Informally, asserting that "X is true" is usually just another way to assert X itself. When I say, "I believe that the Riemann hypothesis is true," I just mean that I believe that all the non-trivial zeros of the Riemann zeta-function lie on the critical line. (Note in particular that I'm not claiming to have a proof of the Riemann hypothesis!) This insight is due to Tarski. If you know what a mathematical statement X asserts, then "X is true" states no more and no less than what X itself asserts. Now, there is a slight caveat here: Mathematicians being cautious folk, some of them will refrain from asserting that X is true unless they know how to prove X or at least believe that X has been proved. So in some informal contexts, "X is true" actually means "X is proved." As we would expect of informal discourse, the usage of the word is not always consistent.
The word "true" can, however, be defined mathematically. Truth is a property of sentences. If you have defined a formal language $L$, such as the first-order language of arithmetic, then you can define a sentence $S$ in $L$ to be true if and only if $S$ holds of the natural numbers. So for example the sentence $\exists x: x > 0$ is true because there does indeed exist a natural number greater than 0. Here it is important to note that true is not the same as provable. The formal sentence corresponding to the twin prime conjecture (which I won't bother writing out here) is true if and only if there are infinitely many twin primes, and it doesn't matter that we have no idea how to prove or disprove the conjecture.
Now, perhaps this bothers you. Is it legitimate to define truth in this manner? Some people don't think so. However, note that there is really nothing different going on here from what we normally do in mathematics. When we were sitting in our number theory class, we all knew what it meant for there to be infinitely many twin primes. Why should we suddenly stop understanding what this means when we move to the mathematical logic classroom? If we understand what it means, then there should be no problem with defining some particular formal sentence to be true if and only if there are infinitely many twin primes. It is as legitimate a mathematical definition as any other mathematical definition.
Now, how can we have true but unprovable statements? And if we had one how would we know? Joel David Hamkins explained this well, but in brief, "unprovable" is always with respect to some set of axioms. Therefore it is possible for some statement to be true but unprovable from some particular set of axioms $A$. In order to know that it's true, of course, we still have to prove it, but that will be a proof from some other set of axioms besides $A$.
