Let me try to answer in some detail the case $p$. So $G$ is a finite group of minimal possible order subject to having a complex irreducible character $\chi$ of degree $p$, where $p$ is a chosen prime. Note Note that $\chi$ is necessarily faithful by minimality of $|G|$. As noted in my comment, by elementary character theory, we necessarily have $|G| >p^{2}$, while also have $p | |G|$, so certainly $|G| \geq p^{2}+p.$ And as noted there, at least when $ p =2$ or a Mersenne prime, there is a Frobenius group $G$ of order $p(p+1)$ with an irreducible character of degree $p$.
Consider first the case that $G = G^{\prime}.$ Then $P \cap Z(G) = P \cap G^{\prime} \cap Z(G) \leq P^{\prime} = 1$ by an elementary transfer argument. Hence $Z(G) = 1$ in this case, using Schur's Lemma, since any non-identity scalar matrix of order prime to $p$ in $G$ would have determinant different from $1$, and would lie outside $G^{\prime}.$ By a Lemma of Burnside (used in the proof of his $p^{a}q^{b}$-theorem), we have $\chi(x) = 0$ for each non-identity $x \in C_{G}(P).$ Thus $\frac{\chi(1)}{|C_{G}(P)|} = \langle {\rm Res}^{G}_{C_{G}(P)}(\chi), 1 \rangle \in \mathbb{Z}$ and $C_{G}(P) = P$ has order $p$. We have already noted that if $N \lhd G$ and $1 \neq N \neq G,$ then $N$ is Abelian, so now $N$ must have order prime to $p$ ( for otherwise $p = \chi(1)$ divides $[G:N]$ by a Theorem of Ito, whereas $[G:N]$ is prime to $p$ when $p$ divides $|N|$, as $|P| = p$). Now $PN$ is non-Abelian, and using Ito's theorem again, we see that the degrees of its irreducible characters divide $p$, so $\chi$ must restrict irreducibly to $PN$, and we must have $PN = G$ by the minimality of $|G|$. But this contradicts $G = G^{\prime}$, since if $G = PN$, we have $G^{\prime} \leq N$ and $G$ is solvable. Our conclusion when $G = G^{\prime}$ is that $G$ is non-Abelian simple and that $p^{2}$ does not divide $|G|$. But then a Theorem of R. Brauer tells us that if $|G| <p^{3}$ for some prime $p$ and $G$ is non-Abelian simple with Sylow $p$-subgroup of order $p >3$, we have $G \cong {\rm PSL}(2,p).$