A finite group of upper triangular matrices has a nilpotent derived group, so a solvable group whose derived group is not nilpotent can't have a faithful representation by upper triangular matrices. One such group is ${\rm GL}(2,3)$ whose derived group is ${\rm SL}(2,3)$, which is certainly not nilpotent (actually $S_{4}$ also works and is a more or less equivalent (but simpler) example).
Later edit: When $k$ has finite characteristic $p >0$, we in fact need $[G,G]$ to be a $p$-group to have any chance of a faithful representation by upper triangular matrices. It is the case that when $G$ is finite and $[G,G]$ is a $p$-group, then $G$ does have a faithful representation by upper triangular matrices over some finite field $k$ of characteristic $p$. For $G/O_{p}(G)$ is then a finite Abelian $p^{\prime}$-group, so all absolutely irreducible representations of $G$ in characteristic $p$ are $1$-dimensional. It follows that there is a finite field $k$ of characteristic $p$ such that the regular representation of $G$ over $k$ (which is certainly faithful) is equivalent as a matrix representation to a representation of $G$ by upper triangular matrices over $k$.
Even later edit: I suppose I should remark for completeness that when $k$ has characteristic $0$, the only finite solvable groups which have a faithful representation by upper triangular matrices with entries in $k$ are Abelian. For if $G$ is such a group, then $[G,G]$ consists of unipotent matrices ( in the chosen faithful representation), and the identity matrix is the only unipotent matrix of finite order in ${\rm GL}(n,k)$ when $k$ has characteristic $0$.