This is correct. It does not follow immediately from the orthogonality relations for group characters, but both they and the above formula follow from the orthogonality relations for individual matrix coefficients with respect to irreducible unitary representations which follow from Schur's Lemma (and were known to Schur). I won't give all details as they can be found in most texts ( and basically generalize to finite dimensional unitary representations of Lie groups), but the basic ideas are: Let $A$ and $B$ be finite dimensional unitary irreducible representations of a finite group $G$ and set $A(g) = [a_{ij}(g)],B(g) = [b_{ij}(g)]$ for each $g \in G.$

For any (suitably sized) matrix $M$, set $T(A,B) = \sum_{g \in G} A(g^{-1})MB(g).$ Then $A(h^{-1})T(A,B)B(h) = T(A,B)$ for all $h \in G,$ and it follows from Schur's Lemma that $T(A,B)$ is the zero matrix if $A,B$ are inequivalent, and that $T(A,B)$ is a scalar matrix if $A,B$ are equivalent.

It follows ( taking different choices for $M$ with exactly one non-zero entry $1$), that we obtain $\sum_{h \in G} a_{ij}(h)\overline{b_{k\ell}(h))} = \frac{|G|}{{\rm dim} A}\delta_{ik}\delta_{j\ell}$ if $A,B$ are equivalent, or always $0$ if $A,B$ are inequivalent.

Now apply these finer relations in the case that $A = B$ affords character $\chi$, and observe that $\chi(h) = \sum_{ i= 1}^{\chi(1)} a_{ii}(h)$ while $\chi(gh) = \sum_{i=1}^{\chi(1)} \sum_{k = 1}^{\chi(1)} a_{ik}(g)a_{ki}(h),$ and you get the formula you want.

Note also that your formula only depends on the character afforded by the representation, so only uses the equivalence type of the representation, so it is fine to choose a unitary representation.