If I understand your conditions correctly, you are assuming that there is a $3$-block of $G$ of defect zero (this corresponds to an irreducible character $\chi$ with $\chi(1)_{3} = |G|_{3}$) and a $2$-block of $G$ of defect $1$ (for in general if a finite group $G$ has order divisibly by $p^{a}$ (but by no higher power of the prime $p$) and $G$ has an irreducible character $\chi$ of degree exactly divisible by $p^{a-1}$ (but by no higher power of $p$), then it is a result of R. Brauer that $\chi$ lies in a $p$-block of defect $1$.
Now since $|G| = 2^{\alpha}3^{\beta}$, we know that $G$ is solvable by Burnside's $p^{a}q^{b}$-theorem. Hence $C_{G}(F(G)) \leq F(G)$, where $F(G)$ is the unique largest nilpotent normal subgroup of $G$.
But for any prime divisor $p$ of $|G|$, $O_{p}(G)$ is contained in each defect group of each $p$-block of $G$, where $O_{p}(G)$ is the largest normal $p$-subgroup of $G$.
Hence we have $O_{3}(G) = 1$ and $|O_{2}(G)| \leq 2$. Since $G$ is a $\{2,3\}$-group, this implies that $F(G)$ has order dividing $2$, so that $F(G) \leq Z(G).$ Since $C_{G}(F(G)) \leq F(G)$, this forces $G = F(G)$ and $|G| \in \{1,2\}$.
So the answer to your question is a resounding "yes".