Let $(\delta_g)_{g\in G}$ be the canonical basis of $\mathbf{C}G$. Define $\exp(g)=\sum\frac1{n!}\delta_g^n$ as an element of the reduced $C^*$-algebra.
I claim that $\exp(g)\in\mathbf{C}G$ iff $g$ has finite order.
It's clear if $g$ has finite order. For the converse, I claim that more generally that for every summable sequence $(c_g)_{g\in G}$, the element $\sum_g c_g\delta_g$ has norm $\ge \left(\sum_g|c_g|^2\right)^{1/2}$; in particular it belongs to $\mathbf{C}G$ (if and) only if $(c_g)$ is finitely supported.
Indeed, for $\varepsilon>0$ (say $<1$), there exists a finite subset $I$ of $G$ such that $\sum_{g\in J}|c_g|\le\varepsilon$ for every finite subset $J$ of $G$ disjoint of $I$. Hence $\left\|\sum_g c_g\delta_g\right\|\ge \left\|\sum_{g\in I} c_g\delta_g\right\|-\varepsilon$, and also $\left\|\sum_{g\in I} c_g\delta_g\right\|_2\ge \left\|\sum_{g} c_g\delta_g\right\|_2-\varepsilon$.
Evaluation at $\delta_1\in\ell_2(G)$ shows that the $C^*$-norm on $\mathbf{C}G$ is $\ge$ than the $\ell^2$ norm. Hence $$\left\|\sum_g c_g\delta_g\right\|\ge \left\|\sum_{g\in I} c_g\delta_g\right\|_2-\varepsilon\ge \left\|\sum_{g} c_g\delta_g\right\|_2-2\varepsilon;$$
since this holds for all $\varepsilon>0$ one gets $\|x\|\ge \|x\|_2$ for $x=\sum_gc_g\delta_g$, which is the desired inequality.
For the conclusion, if $\sum c_g\delta_g=\sum c'_g\delta_g$ with $(c'_g)$ finitely supported, one has $\sum (c_g-c'_g)\delta_g=0$, and hence $\sum_g|c_g-c'_g|^2=0$, so $c_g=c'_g$ for all $g$. Hence $(c_g)$ is finitely supported as well.