I'm familiar with the definition of the category of vector bundles, but I'm trying to derive it from some first principles about general fiber bundles. My intuition is that vector bundles should be understood as Cartan geometries (without a connection) of type $(\mathrm{Aff}(V),\mathrm{GL}(V))$. Starting with the definition of a bundle morphism I would like to understand how to properly constrain that definition in the context of a specific Cartan geometry $(G,H)$ in such a way that morphisms are represented, for example, by linear maps on fibers. I would like to avoid mentioning principal bundles in this process.

1. The "default" structure group of a fiber bundle $E\overset{\pi}{\to}M$ with model fiber $F$ is $\mathrm{Diff}(F)$. In an atlas $\{(U_\alpha,\chi_\alpha)\}$ consisting of maps $\chi_\alpha:\pi^{-1}(U_\alpha)\to U_\alpha\times F$ we have transition functions $t_{\beta\alpha}:U_{\alpha\beta}\to \mathrm{Diff}(F)$ so that $\chi_\beta\circ\chi_\alpha^{-1}(p,f)=(p,t_{\beta\alpha}(p)\cdot f)$.
2. A bundle morphism between two bundles $(E'\overset{\pi'}{\to}M',F)$ and $(E'\overset{\pi'}{\to}M',F')$ consists of two maps $\tilde{h}:E\to E'$ and $h:M\to M'$. Picking atlases $\{(U_\alpha,\chi_\alpha)\}$ and $\{(U_a',\chi_a')\}$ for these two bundles, we can represent the morphism by local maps $\hat{h}_{\alpha a}:(U_\alpha\cap h^{-1}(U_a'))\times F\to F'$ in the following way: $$\tilde{h}(\chi_\alpha^{-1}(p,f))=\chi_a^{\prime -1}(h(p),\hat{h}_{\alpha a}(p,f)).$$
3. By combining these two definitions and writing out the definition of $\hat{h}_{\beta b}$ we get the transformation rule $$\boxed{\hat{h}_{\beta b}(p,t_{\beta\alpha}(p)f)=t_{ba}'(h(p))\hat{h}_{\alpha a}(p,f).}$$

Question: Now if I assume a choice of a $G$-structure on the first bundle and a $G'$-structure on the second, is there a way to simplify this transformation rule so as to turn it into a functor? More precisely, suppose we have two objects $(E\overset{\pi}{\to}M,G\curvearrowright F,\mathcal{G})$ and $(E'\overset{\pi'}{\to}M',G'\curvearrowright F',\mathcal{G}')$, where $\mathcal{G}$ is a $G$-structure on $E$ and $\mathcal{G}'$ is a $G'$-structure on $E'$. Furthermore, choose a map $h:M\to M'$, a map $\psi:F\to F'$, and a homomorphism $\lambda:G\to G'$. Is there a way of defining a map $\tilde{h}:E\to E'$ so that it is a bundle morphism which, in some sense, respects the given structures?

Attempt 1: Given the data that I provided, it would seem that we should be able to express any "reasonable" bundle morphism as $$\hat{h}_{\alpha a}(p,f)=\lambda(\bar{g}_a(p))\psi(g_\alpha(p)f)$$ using some maps $g_\alpha,\bar{g}_a:U_\alpha\cap h^{-1}(U_\alpha')\to G$. The transformation law then becomes $$\psi(f)=\lambda(\bar{g}_b)t_{ba}'(h(p))\lambda(\bar{g}_a)^{-1}\psi\left((g_\beta t_{\beta\alpha}g_\alpha^{-1})^{-1}f\right),$$ where all of the omitted arguments are $p$. Now, this turns out to quickly fail: consider for example isomorphisms of bundles, in which case $h=\mathrm{id}_M$, $\psi=\mathrm{id}_F$ and $\lambda=\mathrm{id}_G$, and we can also take $\alpha=a,\beta=b$ and $\bar{g}_a=e$. Then we get $$t_{\beta\alpha}'=g_\beta t_{\beta\alpha}g_\alpha^{-1}.$$ This is indeed what bundle isomorphisms generally look like, but it doesn't let us make any inferences about what $g_\alpha$'s must be.

Attempt 2: Now let's take some inspiration from Cartan geometry, where the model fiber is actually a homogeneous space $F=G/H=\{Hg\}_{g\in G}$ and the structure group is $H$. This means that there is a transitive left action of $G$ with stabilizers isomorphic to $H$: $G\curvearrowright F$. Then $t_{\beta\alpha}$'s can be assumed to take values in $H$, whereas $g_\alpha$'s can still take values in all of $G$. In this case the last formula obtained for bundle isomorphisms gets replaced with $$t'_{ba}g_\alpha t_{\beta\alpha}^{-1}g_\beta^{-1}\in\mathrm{Stab}_G(f),\text{ for all }f\in F$$ which should somehow imply that $g_\alpha$'s actually commute with some subgroup of $G$ (the group of translations in the vector bundle case), but I don't see how that would happen.

Analysis: my approach so far gives me absolutely no information about what the values of $g$ should actually be, namely elements of a $\mathrm{GL}_k$ subgroup of $G=\mathrm{Aff}_k$. My goal is to come up with a definition of morphisms between bundles with $\mathrm{Aff}$-structures such that it implies that these morphisms must be equivariant with respect to the normal subgroup $\mathbb{R}^k$ of translations, i.e. actual fiberwise linear maps. Perhaps I need to be smart about requiring equivariance of $\psi$, but it's not clear how to do that considering that we need to be able to have different Lie groups acting on the two bundles (e.g. imagine embedding a vector bundle into another vector bundle of another rank, in which case $(G,H)\neq(G',H')$).