I believe this is trivially false: there are families over a compact Kähler base having no singular fibers yet with nontrivial, finite monodromy group. The simplest example I know of is an isotrivial family where $B$ is an elliptic curve and the fibers $X_b$ are hyperelliptic curves.

I believe this is trivially false: there are families over a compact Kähler base having no singular fibers yet with nontrivial, finite monodromy group. The simplest example I know of is an isotrivial family where $B$ is an elliptic curve and the fibers $X_b$ are hyperelliptic curves.

Edit. Although the OP does not specify this in the statement, I suspect the OP is interested in cases where the local monodromies generate the global monodromy group. Obviously if the global monodromy group is finite, then every local monodromy is a finite order element in $\textbf{GL}_n(\mathbb{Z})$. The only such element that is "neat" is the identity element. If every local monodromy is the identity, and if the local monodromies generate the full monodromy group, then also the global monodromy group is trivial.