I don't think there's a general method of approaching this.There There are several methods of obtaining all the Abelian groups of order n for specific n-such as the finite groups of prime order are all cyclic and therefore Abelian,etc. Unless n is very large,proceeding proceeding in this manner will usually get the number of groups down to a managablemanageable size and what remains should be non-Abelian.
I think that's the best you can do unless you want to add the condition that the remaining groups are simple-in which case,a a Sylow analysis would be appropriate and would considerably simplify things.
There are a few general results on non-Abelian groups-like a finite group G is nonAbelian if there are 2 elements in G whose commutator is nontrivial i.e. not the identity. But I don't know if you can use these kinds of results to get the kind of general formula you want-I don't think you can,although although I could be wrong on this.
The best discussions I know of such matters can be found in Herstien'sHerstein's Topics in Algebra,2nd edition and I.Martin Issacs' Finite Group Theory.