I think there is another "degenerate" way to deal with the question as posed, for a general finite group $G$ (later note: which may be developed into more interesting solutions, as we have done in later edits). The key is the connection with idempotents (not necessarily central) of the group algebra $\mathbb{C}G.$
When $G$ is Abelian, the group algebra $\mathbb{C}G$ is clearly commutative, so all idempotents are central. Hence the case of non-central idempotents only arises when $G$ is non-Abelian).
Notice that if such functions $t_{g}(x)$ exist for $ g \in G$ and $x \in \mathbb{C}$, then we must have $t_{g}(0) = t_{g}(0+0) = \sum_{h \in G} t_{gh^{-1}}(0)t_{h}(0)$.
This means that the element $T$ of $\mathbb{C}G$, defined by $T = \sum_{g \in G} t_{g}(0)g$, satisfies $T^{2} = T$, so is either idempotent or zero.
Going in the other direction (ignoring the trivial case $t_{g}(x) = 0$ for all $g,x$), suppose that we are given an idempotent $E = \sum _{g \in G} \lambda_{g}g$
of $\mathbb{C}G.$ Then we may define $t_{g}(x) = e^{x}t_{g}(0) = e^{x}\lambda_{g}$ for all $g \in G$ and all $x \in \mathbb{C}$.
Then for any $g \in G$ and $x,y \in \mathbb{C}$, we have
\begin{eqnarray*}
\sum_{h \in G} t_{gh^{-1}}(x)t_{h}(y) & = & e^{x+y} \sum_{h \in G} \lambda_{gh^{-1}} \lambda_{h} = e^{x+y} \lambda_{g} \\
& = & e^{x+y}t_{g}(0) = t_{g}(x+y),
\end{eqnarray*}
using the fact that $E^{2} = E.$
One obvious case is when $E = \frac{1}{|G|} \left( \sum_{g \in G} g \right)$, which gives rise to the case that $t_{g}(x) = \frac{e^{x}}{|G|}$ for all $g \in G, x \in \mathbb{C}$ .
Another easy case is when $E = 1_{G}$, which gives rise to the case $t_{g}(x) = e^{x}$ if $g = 1_{G}, 0$ otherwise (for $g \in G, x \in \mathbb{C}$).
A more interesting example when $G = S_{3}$ and $E$ is the non-central idempotent
$ \frac{1+(12)}{2} - \frac{1}{6} \left(\sum_{ \sigma \in S_{3}} \sigma \right)$,
in which case ( for any $x \in \mathbb{C}$), we have $e^{-x}t_{g}(x) = \frac{1}{3}$ for $g = 1_{G}$ or $g = (12)$,
and $e^{-x}t_{g}(x) = \frac{-1}{6}$ otherwise.
This method gives existence of some solutions to the convolution equations, but there are other types of solution, as you already showed. In the solutions above we can also replace $e^{x}$ by $e^{\alpha x}$ for a fixed constant $\alpha$ (independent of $x$).
Later edit: Let us try to develop these arguments further and obtain more information about general solutions to the convolution equations.
Suppose that we solutions $\{t_{g}(x) : g \in $G$ \}$ for the convolution equations. For each $x \in \mathbb{C}$, let $E(x) = \sum_{g \in G} t_{g}(x) g$ in the group algebra $\mathbb{C}G$. We have seen that $E(0)$ is an idempotent of $\mathbb{C}G$.
The convolution equations imply that $E(x) = E(0 + x) = E(0)E(x)$ and $E(x) = E(x+0) = E(x)E(0)$ for each $x \in \mathbb{C}$. Hence $E(x)$ always lies in the subalgebra $E(0)\mathbb{C}G E(0)$ of the group algebra $\mathbb{C}G$, and this subalgebra is isomorphic to the endomorphism algebra of the right $\mathbb{C}G$-module $E(0)\mathbb{C}G$. Also, $E(0)$ is the identity element of this algebra. Also, the convolution equations imply that
$E(x)E(y) = E(x+y) = E(y+x) = E(y)E(x)$ for all $x,y \in \mathbb{C}$.
We also remark that $E(x)$ is a unit in the algebra $E(0)\mathbb{C}G E(0)$, and that its inverse is $E(-x)$.
Let us consider the case that $E(0)$ is a primitive idempotent of $\mathbb{C}G$. In that case, Schur's Lemma tells us that $E(0)\mathbb{C}GE(0)$ consists of scalar multiples of $E(0)$.
In that case, for each $x \in \mathbb{C}$, we have $E(x) = \mu_{x}E(0)$ for some complex number $\mu_{x}$, and by the remarks above, we have $\mu_{x}\mu_{y} = \mu_{x+y}$ for all $x,y \in \mathbb{C}$. This forces $\mu_{x} = e^{\alpha x}$ for each $x \in \mathbb{C}$, for some constant $\alpha.$
Hence the only solutions for which $E(0)$ is a primitive idempotent of $\mathbb{C}G$ are the "degenerate" ones described earlier.
This indicates how to deal with the case that $E(0)$ is a general idempotent of $\mathbb{C}G$. We may decompose $E(0)$ as a sum of mutually orthogonal primitive idempotents of $\mathbb{C}G$, say
$E(0) = \sum_{j= 1}^{t} E_{j}(0)$, where $E_{i}(0) E_{j}(0) = \delta_{ij} E_{i}(0)$ for each $i,j.$
In the case that no two of the $E_{i}(0)$ are conjugate via a unit of $\mathbb{C}G$, the situation is reasonably clear. For then we see that
$E(0) \mathbb{C}G E(0)$ is a commutative algebra which is the direct sum of the $1$-dimensional algebras $E_{j}(0)\mathbb{C}G E_{j}(0)$. Then we may see that there are complex constants $a_{j} : 1 \leq j \leq t$ such that
$ E(x) = \sum_{j=1}^{t} e^{a_{j}x} E_{j}(0).$
Even later edit: In comments, it is asked what is necessary if we insist the determinant ${\rm det}[t_{gh^{-1}}(x)]$ is equal to $1$ for all $x$. We illustrate with one example. When $G = \langle u \rangle$ is cyclic of order $2$, there are just two mutually orthogonal idempotents in the group algebra $\mathbb{C}G$. These are $ E_{1} = \frac{1_{G} + u}{2} $ and $E_{2} = \frac{1_{G} - u}{2}. $
Then the above methods give a general solution to the convolution equations of the form $ \sum_{g \in G}t_{g}(x)g = e^{c_{1}x} E_{1} + e^{c_{2}x}E_{2}.$
This gives the determinant ${\rm det}[(t_{gh^{-1}}(x))])$ to be $e^{c_{1}x+c_{2}x}$. Hence this determinant is $1$ for all $x$, if and only if $c_{2} = -c_{1}.$
Incidentally, to partially address one question of the OP in comments, this example for the cyclic group of order two may be ``lifted" to a solution of the convolution equations for $G = S_{n}$ for any $n$. I don't give all details, but we may choose constants $c_{1}$ and $c_{2}$ and set
$t_{g}(x) = \frac{1}{n!}\left( e^{c_{1}x} + e^{c_{2}x} \right)$ whenever $g$ is an even permutation and $t_{g}(x) = \frac{1}{n!} \left( e^{c_{1}x} - e^{c_{2}x} \right)$ if $g$ is an odd permutation. However, we then find that the matrix $[t_{gh^{-1}}(x)]$ is singular for all $x$ when $n > 2.$
Returning to the general solutions, in the case that two or more of the $E_{j}(0)$ are conjugate via a unit of the group algebra $\mathbb{C}G$, the analysis is more subtle.
Latest edit (also addressing some issues in comments): Let us recall that for any finite group $G$, the group algebra $\mathbb{C}G$ is isomorphic to a direct sum of (mutually annihilating) full matrix algebras $ \bigoplus_{\chi} M_{\chi(1)}(\mathbb{C})$, where $\chi$ runs over the complex irreducible characters of $G$. This gives a decomposition of $1_{G}$ as a sum of mutually orthogonal primitive idempotents
that is $1_{G} = \sum_{\chi} \sum_{j = 1}^{\chi(1)} E_{j}(\chi)$. Here, $E_{j}(\chi)$ is represented by an idempotent matrix of trace $1$ in any irreducible representation of $G$ which affords $\chi$, and is represented by $0$ in any irreducible representation of $G$ affording an irreducible character $\mu \neq \chi.$ Also, for each $\chi$, we have $\sum_{j=1}^{\chi(1)} E_{j}(\chi)$ is a central idempotent of $\mathbb{C}G$, and acts as the identity in any irreducible representation of $G$ affording character $\chi$.
This decomposition of $1_{G}$ into a sum of mutually orthogonal primitive idempotents is not quite unique in general ( but it is when $G$ is Abelian). In general, it is unique up to conjugation by a unit of $\mathbb{C}G$ (and reordering the idempotents).
For non-commutative $G$, we now outline how to construct some solutions to
the convolution equations which have ${\rm det}([t_{gh^{-1}}(x)]) = 1$ for all $x$, but the list is not exhaustive in general.
We take a set of pairwise distinct (this is not really necessary, but is useful for purposes of exposition) non-zero complex constants $\{ c_{j}(\chi) : \chi \in {\rm Irr}(G), 1 \leq j \leq \chi(1) \}$ with the property that
$\sum_{\chi,j} \chi(1) c_{j}(\chi) = 0.$ We further insist that $\{ e^{c_{j}(\chi)} : \chi \in {\rm Irr}(G), 1 \leq j \leq \chi(1) \}$ still consists of $\sum_{\chi} \chi(1)$ distinct elements. This is always possible to achieve for non-trivial $G$. For example, we may take all but one of the $c_{j}(\chi)$ to be real, positive and pairwise distinct ( with distinct exponentials, all real and greater than one). The zero sum condition then forces the choice of the remaining $c_{j}(\chi)$, and it clearly must be negative with negative exponential.
It is convenient to introduce exponentials of elements of the group algebra $\mathbb{C}G.$ To do this, we take an element $A$ of $\mathbb{C}G$, and take its component $A_{\chi}$ in the matrix algebra summand $M_{\chi}(\mathbb{C})$. Then we define $\exp(A)$ to be the unique element of $\mathbb{C}G$ which has component $\exp(A_{\chi})$ ( the usual matrix exponential) in $M_{\chi}(\mathbb{C}).$ As usual, if $A$ and $B$ are commuting elements of $\mathbb{C}G$, then we have $\exp(A+B) = \exp(A)\exp(B).$
Now for any complex number $x$, we define the element $E(x)$ to be
$\exp{\left(\sum_{\chi} \sum_{j = 1}^{\chi(1)}c_{j}(\chi)x E_{j}(\chi) \right)}.$
Then we see that whenever $\sigma$ is an irreducible representation of $G$ affording character $\mu$, $\sigma(E(x))$ is a diagonalizable matrix with
distinct eigenvalues $e^{c_{j}(\chi)x} : 1 \leq j \leq \chi(1)$ (strictly speaking, we need to avoid the case that $x$ has the form $\frac{2\pi i}{c_{r}-c_{s} }$ for any $r \neq s$).
It follows (with the noted exceptions) that in the regular representation of $G$, $E(x)$ has the eigenvalue $e^{c_{j}(\chi)x}$ with multiplicity $\chi(1)$ for $1 \leq j \leq \chi(1)$, for each irreducible $\chi$.
Since we assumed that $\sum_{\chi} \chi(1) \left( \sum_{j=1}^{\chi(1)} c_{j}(\chi) \right) = 0$, we see that the determinant of the image of
$E(x)$ in the regular representation is $\prod_{\chi}\left( \prod_{j = 1}^{\chi(1)} e^{c_{j}(\chi)x} \right)^{\chi(1)} = 1$ for all $x$.
But writing $E(x) = \sum_{g \in G} t_{g}(x) g$ in the group algebra $\mathbb{C}G$, we see that $E(x)$ is represented by the matrix $[t_{gh^{-1}}(x)]$ in the regular representation.
Before treating the explicit case $G = S_{3}$, we remark that in general it is not straightforward to determine explicit formulae for the primitive idempotents of the group algebra $\mathbb{C}G$ when $G$ is a non-Abelian group. There is an explicit general procedure due to A. Young when $G$ is the symmetric group $S_{n}.$ However, for the case $G = S_{3}$, we may proceed directly.
We know from the theory described above that there should be four mutually orthogonal primitive idempotents of the group algebra $\mathbb{C}G$ when $G$ is the symmetric group $S_{3}.$ Two of these are easy, but we also need to decompose the central idempotent corresponding to the degree $2$ irreducible character. This idempotent is $ X = \frac{3(1)- [(1) +(123) +(132)]}{3}.$
We have written the idempotent in this fashion to make it clear that it commutes with the idempotent $Y = \frac{(1) + (12)}{2}.$ Then we see easily that $XY = YX = \frac{(1) + (12)}{2} - \frac{ \sum_{g \in G} g}{6}.$
This is a primitive idempotent of $\mathbb{C}G$, and may serve as our idempotent $E_{1}(\chi)$, where $\chi$ is the degree $2$ irreducible character. The primitive idempotent $E_{2}(\chi)$ is then easily seen to be
$\frac{(1)-(12)}{2} - \frac{ \sum_{g \in G} {\rm sign}(g) g}{6}.$
Hence we may set $E_{1}( \lambda_{0}) = \frac{\sum_{g \in G} g }{6}$,
$E_{1}( \lambda_{1}) = \frac{\sum_{g \in G} {\rm sign}(g)g }{6}$,
where $\lambda_{0}$ is the trivial character, $\lambda_{1}$ is the sign character.
Then we may choose any four non-zero complex constants $c_{0},c_{1},c_{2},c_{3}$ with $c_{0} + c_{1} + 2(c_{2}+c_{3}) = 0$
and set $E(x) = e^{c_{0}x} E_{1}(\lambda_{0}) + e^{c_{1}x}E_{1}(\lambda_{1}) + e^{c_{2}x}E_{1}(\chi) + e^{c_{3}x}E_{2}(\chi)$.
This leads to $t_{1}(x) = \frac{e^{c_{0}x}}{6} + \frac{e^{c_{1}x}}{6} +
\frac{e^{c_{2}x}}{3} + \frac{e^{c_{3}x}}{3}$, $t_{(12)}(x) = \frac{e^{c_{0}x}}{6} - \frac{e^{c_{1}x}}{6} +
\frac{e^{c_{2}x}}{3} - \frac{e^{c_{3}x}}{3}$, etc. , producing a solution to the convolution equations with ${\rm det}([t_{gh^{-1}}(x)]) = 1$ for all $x$.
