A nice way to understand this is to give a "better" characterization of polarizations that avoids any reliance on structures that only are available on geometric fibers. The claim is that a homomorphism $\lambda:A \rightarrow A^t$ is a polarization if and only if it satisfies the following conditions: (i) $\lambda$ is symmetric with respect to double duality, (ii) the line bundle $(1, \lambda)^{\ast}(\mathscr{P}_A)$ on $A$ is fiberwise ample. This characterization deserves to be more widely known, and it is a bit surprising that (as far as I am aware) it is not mention in any of Mumford's papers or books.
[We are assuming that $A$ is projective Zariski-locally on $S$ so that it admits a dual abelian scheme by Grothendieck's work, or one appeals to deeper methods relying on algebraic spaces to make the dual (as a scheme, not just algebraic space) in general.]
By rigidity considerations (after passing to the case of a noetherian base, by a small argument) whether or not (i) holds is checkable on geometric fibers, and likewise for (ii), so to prove such an equivalence it suffices to check for abelian varieties over an algebraically closed field.
If $\lambda = \phi_L$ for some line bundle $L$ on $A$ then the symmetry of $\lambda$ is part of the classical theory, in which case $(1, \lambda)^{\ast}(\mathscr{P}_A) \in L^{\otimes 2} \cdot A^t(k)$ inside ${\rm{Pic}}(A)$. But ampleness or not of a line bundle on such $A$ only depends on the line bundle class in ${\rm{Pic}}(A)/A^t(k)$, so the condition (ii) is equivalent to ampleness of $L^{\otimes 2}$, which in turn is equivalent to ampleness of $L$ (and then $\phi_L$ is an isogeny by the classical theory too).
To complete the proof of the equivalence, it suffices to show conversely that every symmetric homomorphism $\lambda$ arises as $\phi_L$ for some line bundle $L$ on $A$. This is a consequence of Theorem 2 in section 20 and Theorem 3 in section 23 of Mumford's book on abelian varieties, along with consideration of Weil pairings. (Poonen and Stoll gave examples of smooth projective geometrically connected curves over $\mathbf{Q}$ for which the principal polarization of the Jacobian $J$ does not arise as $\phi_L$ for any $L$ on $J$.)
Having established the equivalence, it suffices to show that each of (i) and (ii) corresponds to an open condition on the base. More specifically, for (i) it suffices to show that if a homomorphism $f:A \rightarrow B$ between abelian schemes (e.g., $\lambda - \lambda^t$) vanishes on fibers at some $s \in S$ then it vanishes over an open neighborhood of $s \in S$, and for (ii) it suffices to show that if $X \rightarrow S$ is a proper finitely presented map of schemes and $L$ is a line bundle on $X$ then the locus $U$ of $s \in S$ such that $L_s$ is ample on $X_s$ is open in $S$. The former is a consequence of rigidity (after passing to the case when the base is local with $s$ as its closed point, and then reducing to the case when the local base ring is noetherian), and the latter is EGA IV$_3$, 9.6.4 (and by IV$_3$ 9.6.5 the restriction of $L$ to $X_U$ is relative ampleness over $U$ -- i.e., ample over every affine open in $U$, quite remarkable without any flatness hypotheses and absolutely crucial for applications to the use of ampleness for proving effective descent in the context of moduli problems).
In fact, being a polarization is a (finitely presented) closed condition on $S$ too. We may reduce to the case of noetherian $S$, and then (i) this is again a consequence of rigidity considerations, whereas for (ii) it lies deeper: by the valuative criterion for properness (applied to open immersion in the base) it suffices to consider the case of abelian schemes over a discrete valuation ring, but one has to do a fair bit more work to handle that; this is discussed early in the book of Faltings and Chai.