These manifolds have actually been studied to some extent by Campana and Peternell in their series of papers "Towards a Mori theory on compact Kähler threefolds I, II, III". In those papers they mention a folklore conjecture (which can be found for example here), that says that every simple Kähler manifold of odd dimension $n>1$ must be Kummer (i.e. bimeromorphic to the quotient of a complex torus by a finite group).
In the paper number II Peternell shows that this conjecture in dimension $3$ follows from MMP plus abundance for Kähler threefolds. On the other hand in paper number III he shows that abundance does hold for Kähler threefolds with the possible exception of simple non-Kummer manifolds (which should not exist).
On the other hand more recent developments using model theory seem to suggest that in the even-dimensional case apart from Kummer manifolds the only other simple Kähler manifolds are in generically finite-to-finite correspondence with an irreducible hyperkähler manifold, like in Misha's answer.
Anyway, even after all this work it's unclear whether simple non-Kummer odd-dimensional manifolds exist or not. It's certainly an interesting problem, but probably a very hard one.
