Where does this problem come from? Why would you hope for something like this? I am not trying to be snarky, I am just wondering if there is a **true** statement that might better fit your situation.

At any rate, counterexamples are ~~easy~~ possible to produce via varieties with a group action such that the variety deforms, but the group action is rigid. Here is one construction. Let $Y$ be a curve of genus $g\geq 2$ with an action of a finite group, $$\mu:G\times Y \to Y, $$ such that (i) the pair $(Y,\mu)$ is rigid, and (ii) there are no points of $Y$ that are fixed by the entire group $G$. In characteristic $0$, the second condition is automatic if $G$ is noncyclic (in characteristic $p$, it is true if the quotient by the maximal quasi-$p$-group is noncyclic), and the first condition is automatic if the quotient map, $Y\to Y/G$, is a map to a rational curve branched over precisely $3$ points. The Klein quartic with its full group of automorphisms is one example.

Now let $X'$ be the product $$X' = \prod_{g\in G} Y = \text{Hom}_{\text{Sets}}(G,Y).$$
In other words, $X'$ is the projective $k$-scheme of ordered tuples $x'=(y_g)_{g\in G}$ of elements $y_g\in Y$. There is a projection $\text{pr}_1:X'\to Y$ sending $x'$ to $y_1$.

There is a diagonal embedding $$\Delta : Y \to X', \ \ y \mapsto (y)_{g\in G},$$
and the action $\mu$ also induces a second embedding $$\Gamma_\mu:Y \to X', \ \ y \mapsto (\mu(g,y))_{g\in G}.$$ By hypothesis (ii) above, the images $\Delta(Y)$ and $\Gamma_{\mu}(Y)$ are disjoint. Denote the blowing up of $X'$ along the closed subscheme $\Delta(Y)\sqcup \Gamma_{\mu}(Y)$ by $\nu$, $$\nu:X\to X'.$$
Define $f$ to be $\text{pr}_1\circ \nu$.

Every fiber is a product of smooth projective curves blown up at two points, and there are deformations of all of these. The base is just $Y$, and this deforms as well. However, the minimal model / canonical model / Albanese image of $X'$ is $X$. Every deformation of a product of curves is a product of curves, cf. the work of van Opstall. The fundamental locus of the Albanese morphism is a disjoint union of two curves. But for either curve, every projections of the curves to a curve factor is an isomorphism. So you can use one of the two fundamental curves, e.g., the curve $\Delta(Y)$, to identify all curve factors with the fundamental curve. Then the second fundamental curve gives a sequence of automorphisms of that curve. Now use that $(Y,\mu)$ is rigid to conclude that $X$ is rigid.