Let us imagine we are building the Gell-Mann matrices. For us, SU($n$) is the group of $n \times n$ unitary matrices with determinant 1. An element $U$ can be written $U = e^{i \alpha_i T_i}$, where $i = 1,\ldots,n^2-1$ and the $\alpha_i$ are real. The $T_i$ are a basis for the algebra. They must be traceless and hermitian. Consider $\mathrm{tr}\ T_i T_j$. This matrix is real and symmetric. Diagonalize it. Call your new basis $\lambda_i$. Normalize so $\mathrm{tr}\ \lambda_i \lambda_j = 2\delta_{i j}$. From here the Gell-Mann matrices are further specified by the condition that the Pauli matrices sit nicely inside, etc.

Why diagonalize $\mathrm{tr}\ T_i T_j$? Of course, it is because we want a nice orthogonal basis. Would you rather have as a basis for two space {(1,2),(3,4)} or {(1,0),(0,1)}? The orthogonal representation is also simply related to the raising and lowering operators used to build reps and decompose product reps.
Notice that we could choose another basis $X_i = M_{i j}\lambda_j$ where $M$ is invertible and real.
This does not change the algebra but, for example, the structure constants will be different, the relation of the basis to the raising and lowering operators will be complicated, and the $X_i$ will not be orthogonal in general.