By Zariski density arguments, you may replace $PU(d)$ by $PGL(d)=SL(d) \quad modulo \quad centre$. By the Borel -Weil Theorem,  every irreducible representation of $SL(d)$ is of the form $V_{\chi}=Ind_B ^G (\chi)$ where $B$ is the Borel subgroup of upper triangular matrices, and $\chi$ is an anti-dominant character. In order that this representation descend to $PGL(d)$ it is necessary and sufficient that $\chi$ be trivial on the centre of $SL(d)$. 

By Using the Weyl dimension formula, one can then see that the smallest $V_{\chi}$  is  the adjoint representation. 

[Marc Palm's $d$th symmetric power has dimension $d^d$ which is too large].  

I could give more details of the proof that the adjoint rep is the right one, but the "margin  is too small".