Note that $\text{Inn}(X)$ is isomorphic to $X/Z(X)$. So your requirement is equivalent to $X$ being simple modulo the centre. For example $SL_n(\mathbb F_{p^m})$ satisfies this, since the centre is given by the scalars and $PSL_n(\mathbb F_{p^m})$ is simple.
More generally, central extensions of a given group $S$ (which you take to be simple) by the group $A$ are classified by $H^2(S,A)$. So if you fix $S$ and $A$, you can try computing this cohomology group and thereby deciding if there are any such extensions apart from the direct product. Note that while $H^2$ actually classifies extensions up to splitting, in this particular case it just classifies extensions up to your trivial example, since a semidirect product by the centre is a direct product.
A class of examples of the sort you are looking for is given by so-called quasi-simple groups. They are exactly the sorts of groups you want but with the additional requirement that $X'=X$. This is not so severe: indeed, already your requirement implies that $X'Z/Z \triangleleft X/Z$, which is simple. So either $X'\leq Z$ and then $X/X'$ is simple and abelian, hence $X$ is cyclic modulo the centre, hence abelian, or $X'Z = X$. So you are never too far away from $X'=X$. (I suspect that one can show that your groups are either quasi-simple or direct products. Proofs or counterexamples will be gratefully received.)