Since you have read Rudin's "Real and complex analysis", you are ready to attack Rudin's "Fourier analysis on groups", which is equally pleasant reading.

Still valuable for the link with Banach algebra theory, is L.H. Loomis' "An introduction to abstract harmonic analysis".

For connections with unitary representations: the 2nd half of Dixmier's "C*-algebras", or better R. Howe and E.C. Tan "Non-abelian harmonic analysis (applications of $SL_2(\mathbb{R})$)" (everything is in the subtitle!)

If you want to see connections with number theory, I recommend Weil's "Basic number theory".

Now you can guess my age from this list of references!