We have an odd number of colours. There is an edge between any pair of colours, or else those colours could be combined. So we can easily generate a matching with $\frac{\chi - 1}{2}$ edges. We are given that we cannot add an edge.

Take the colour $C$ not currently involved in any matching. Suppose that some vertex $v \in C$ is not adjacent to every other colour. Then let $v$ be adjacent to some vertex of colour $A$, but not to any vertex of colour $B$. In the initial matching, pair $A$ with $B$, to get a matching of $\frac{\chi -1}{2}$. Now delete the edge between $A$ and $B$ and add the $v$ to $A$ edge and one $B$ to $C$ edge. Now the matching number is too big. Hence $v$ does not exist - every vertex in $C$ is adjacent to a vertex of every other colour. Since $C$ was arbitrary, every vertex is adjacent to a vertex of every other colour.

Now we simply pick a vertex not yet in the matching and not in the colour $C$ and match it with something in $C$ to break the rule. Hence everything not in $C$ is in the matching, so each colour has only one vertex. But $C$ was chosen arbitrarily, so it too has just one member. Hence the graph is complete.

EDIT: another answer just pointed out that isolated vertices can be admitted, which I missed out when I assumed that the vertex $v$ is adjacent to some other vertex (whose colour I called $A$). Apologies.