Tarski defined what it means to say that a first-order statement is true in a structure $M\models \varphi$ by a simple induction on formulas. This is a completely mathematical definition of truth.

Goedel defined what it means to say that a statement $\varphi$ is provable from a theory $T$, namely, there should be a finite sequence of statements constituting a proof, meaning that each statement is either an axiom or follows from earlier statements by certain logical rules. (There are numerous equivalent proof systems, useful for various purposes.)

The Completeness Theorem of first order logic, proved by Goedel, asserts that a statement $\varphi$ is true in all models of a theory $T$ if and only if there is a proof of $\varphi$ from $T$. Thus, for example, any statement in the language of group theory is true in all groups if and only if there is a proof of that statement from the basic group axioms.

The Incompleteness Theorem, also proved by Goedel, asserts that any consistent theory $T$ extending some a very weak theory of arithmetic admits statements $\varphi$ that are not provable from $T$, but which are true in the intended model of the natural numbers. That is, we prove in a stronger theory that is able to speak of this intended model that $\varphi$ is true there, and we also prove that $\varphi$ is not provable in $T$. This is the sense in which there are true-but-unprovable statements.

The situation can be confusing if you think of provable as a notion by itself, without thinking much about varying the collection of axioms. After all, as the background theory becomes stronger, we can of course prove more and more. The true-but-unprovable statement is really unprovable-in-$T$, but provable in a stronger theory.

Actually, although ZFC proves that every arithmetic statement is either true or false in the standard model of the natural numbers, nevertheless there are certain statements for which ZFC does not prove which of these situations occurs.

Much or almost all of mathematics can be viewed with the set-theoretical axioms ZFC as the background theory, and so for most of mathematics, the naive view equating true with provable in ZFC will not get you into trouble. But the independence phenomenon will eventually arrive, making such a view ultimately unsustainable. The fact is that there are numerous mathematical questions that cannot be settled on the basis of ZFC, such as the Continuum Hypothesis and many other examples. We have of course many strengthenings of ZFC to stronger theories, involving large cardinals and other set-theoretic principles, and these stronger theories settle many of those independent questions. Some set theorists have a view that these various stronger theories are approaching some kind of undescribable limit theory, and that it is that limit theory that is the true theory of sets. Others have a view that set-theoretic truth is inherently unsettled, and that we really have a multiverse of different concepts of set. On that view, the situation is that we seem to have no standard model of sets, in the way that we seem to have a standard model of arithmetic.