Note that $\text{Inn}(X)$ is isomorphic to $X/Z(X)$. So your requirement is equivalent to $X$ being simple modulo the centre, also called "quasi-simple". For example $SL_n(\mathbb F_{p^m})$ satisfies this, since the centre is given by the scalars and $PSL_n(\mathbb F_{p^m})$ is simple.
More generally, central extensions of a given group $S$ (which you take to be simple) by the group $A$ are classified by $H^2(S,A)$. So if you fix $S$ and $A$, you can try computing this cohomology group and thereby deciding if there are any such extensions apart from the direct product.
Googling for quasi-simple will also give you some more explicit examples.