I shall elaborate on the comments and provide a sketch of the proof (for which I unfortunately don't know a reference). As $k$ is a perfect field, $X$ is smooth if and only if $X$ is regular. By a well-known theorem of Serre $X$ is regular
if and only if for every closed point $x\in X$ the local ring $\mathscr{O}_{X,x}$ is of finite homological dimension,
Theorem. The following are equivalent:
(i) $X$ is regular;
(ii) $Coh(X)$ has finite homological dimension;
(iii) $D^b(X)$ is Ext-finite.
Proof. (i)$ \Rightarrow $(ii): Serre duality.
(ii)$ \Rightarrow $(iii): By assumption the condition required by Ext-finiteness holds for all complexes concentrated in one degree, and it follows for all bounded complexes for instance by use of standard spectral sequences (see remark 3.7 in Huybrecht's `Fourier-Mukai transforms').
(iii)$ \Rightarrow $(i): Suppose that $X$ is not regular, let $x\in X$ be a closed singular point. Let $\mathscr{O}_x$ be the skyscraper sheaf with stalk $\kappa(x)$ at $x$. Then the local-to-global spectral sequence yields isomorphisms $\mathrm{Ext}^n_X(\mathscr{O}_x,\mathscr{O}_x)\simeq\mathrm{Ext}^n_{\mathscr{O}_{X,x}}(\kappa(x),\kappa(x))$. The latter is nonzero for infinitely many $n$ because $\mathscr{O}_{X,x}$ has infinite homological dimension. Hence $D^b(X)$ is not Ext-finite.