The representation you're looking at is $\mathrm{Ind}_M^N1$ and as such, its decomposition into irreducibles is very well understood using Kirillov's orbit method. (Essentially, the irreducibles that enter correspond to the coadjoint orbits in the image of the moment map
$T^*(N/M)\to\mathfrak n^*$.)
I'd day say the basic paper on the subject is this one by Corwin, Greenleaf, and Grélaud. It has references to the earlier work by Kirillov himself, and you'll find more in mathscinet's forward references to reviews citing it.

