Are you sure that $G$ isn't required to be connected? I think this is needed in order to construct the "valued root datum" structure which underlies Bruhat-Tits structure theory. Anyway, the key point is that there is the concept of "valuation" on the root datum, which is really a collection of "valuations" on the possibly non-commutative groups $U_a(K)$ subject to axioms defined in the first big Bruhat-Tits paper in IHES, which I'll call BTI. The existence of this kind of structure on $G(K)$ requires the full power of the theory of the 2nd Bruhat-Tits IHES paper (developed in more "modern" terms in later work of others, such as J-K. Yu), and it requires connectedness of $G$. On the set of such "valuations" there is a natural free action of $V$ and elements of the same equivalence class are called "parallel". The equivalence classes are naturally affine spaces for $V$, and the group $N(K)$ acts naturally on the entire set preserving each equivalence classes through an action by affine transformations (with $Z(K)$ acting through the translation formulas as you have written down). This is all pure group theory formalism (but far from trivial to set up), the definitions of which have nothing to do with any topological structure on $K$. The specification of a valuation on $K$ selects out a preferred equivalence class, and that is the one used to define $\phi$.
(For example, if $G = {\rm{SL}}_2$ and $S$ is the diagonal split maximal torus of $G$ over a field $F$ then the parallelism classes of "valuations" on $G(F)$ in accordance with the root datum for $(G,S)$ correspond exactly to choices of nontrivial non-archimedean $\mathbf{R}$-valued valuations on the abstract field $F$.)
So what you're missing is the (highly non-trivial to develop!) definition of the principal homogeneous space for $V$ which supports the action of $N(K)$. In other words, although one can say in concrete terms that the target of $\phi$ is the group of affine transformations of $V$, this is conceptually misleading: it is really the group of automorphisms of a more intrinsic affine space for $V$ in which there is absolutely no canonical base point (intrinsic to $(G,S,\Phi^+,\omega)$). I suppose there could be a way to make the definition of $\phi$ by bare hands (or at least give formulas, without proving things are well-defined), but my understanding (which could be incomplete) is that using a specific parallelism class of of "valuations" as indicated above provides the only natural way to make the definition. Take a look at section 6 of BTI to learn what a valued root datum is, and the many nontrivial properties of this kind of structure. I think that BTI is more illuminating in certain conceptual respects than the Corvallis paper (though of course it doesn't have the rich supply of interesting examples as in the Corvallis paper, and is a rather challenging paper to read).