By Zariski density arguments, you may replace $PU(d)$ by $PGL(d)=SL(d) \quad modulo \quad centre$. By the Borel -Weil Theorem, every irreducible representation of $SL(d)$ is of the form $V_{\chi}=Ind_B ^G (\chi)$ where $B$ is the Borel subgroup of upper triangular matrices, and $\chi$ is an anti-dominant character. In order that this representation descend to $PGL(d)$ it is necessary and sufficient that $\chi$ be trivial on the centre of $SL(d)$.
By Using the Weyl dimension formula, one can then see that the smallest $V_{\chi}$ is the adjoint representation.
[Marc Palm's $d$th symmetric power has dimension $d^d$ which is too large].