I'm familiar with the definition of the category of vector bundles, but I'm trying to derive it from some first principles about general fiber bundles. My intuition is that vector bundles should be understood as Cartan geometries (without a connection) of type $(\mathrm{Aff}(V),\mathrm{GL}(V))$. A $\mathrm{GL}$-structure on a fiber bundle obviously preserves the vector space structure of the model fiber, but I would think it should also force bundle morphisms to be fiberwise linear. Thus, starting with the definition of a bundle morphism I would like to understand how to properly constrain that definition in the context of a specific Cartan geometry $(G,H)$ in such a way that morphisms are represented, for example, by linear maps on fibers. I would like to avoid mentioning principal bundles in this process.

1. The "default" structure group of a fiber bundle $E\overset{\pi}{\to}M$ with model fiber $F$ is $\mathrm{Diff}(F)$. In an atlas $\{(U_\alpha,\chi_\alpha)\}$ consisting of maps $\chi_\alpha:\pi^{-1}(U_\alpha)\to U_\alpha\times F$ we have transition functions $t_{\beta\alpha}:U_{\alpha\beta}\to \mathrm{Diff}(F)$ so that $\chi_\beta\circ\chi_\alpha^{-1}(p,f)=(p,t_{\beta\alpha}(p)\cdot f)$.
2. A bundle morphism between two bundles $(E'\overset{\pi'}{\to}M',F)$ and $(E'\overset{\pi'}{\to}M',F')$ consists of two maps $\tilde{h}:E\to E'$ and $h:M\to M'$. Picking atlases $\{(U_\alpha,\chi_\alpha)\}$ and $\{(U_a',\chi_a')\}$ for these two bundles, and denoting $U_{\alpha a}=U_\alpha\cap h^{-1}(U_a')$, we can represent the morphism by local maps $\hat{h}_{\alpha a}:U_{\alpha a}\times F\to F'$ in the following way: $$\tilde{h}(\chi_\alpha^{-1}(p,f))=\chi_a^{\prime -1}(h(p),\hat{h}_{\alpha a}(p,f)).$$
3. By combining these two definitions and writing out the definition of $\hat{h}_{\beta b}$ we get the transformation rule $$\boxed{\hat{h}_{\beta b}(p,t_{\beta\alpha}(p)f)=t_{ba}'(h(p))\hat{h}_{\alpha a}(p,f).}$$

Question: Now if I assume a choice of a $G$-structure on the first bundle and a $G'$-structure on the second, is there a way to simplify this transformation rule so as to turn it into a functor? More precisely, suppose we have two objects $(E\overset{\pi}{\to}M,G\curvearrowright F,\mathcal{G})$ and $(E'\overset{\pi'}{\to}M',G'\curvearrowright F',\mathcal{G}')$, where $\mathcal{G}$ is a $G$-structure on $E$ and $\mathcal{G}'$ is a $G'$-structure on $E'$. Furthermore, choose a map $h:M\to M'$, a map $\psi:F\to F'$, and a homomorphism $\lambda:G\to G'$. Is there a way of defining a map $\tilde{h}:E\to E'$ so that it is a bundle morphism which, in some sense, respects the given structures?

Attempt 1: Given the data that I provided, it would seem that we should be able to express any "reasonable" bundle morphism as $$\hat{h}_{\alpha a}(p,f)=\lambda(\bar{g}_a(p))\psi(g_\alpha(p)f)$$ using some maps $g_\alpha,\bar{g}_a:U_{\alpha a}\to G$. The transformation law then becomes $$\psi(f)=\lambda(\bar{g}_b)t_{ba}'(h(p))\lambda(\bar{g}_a)^{-1}\psi\left((g_\beta t_{\beta\alpha}g_\alpha^{-1})^{-1}f\right),$$ where all of the omitted arguments are $p$. Now, this turns out to quickly fail: consider for example isomorphisms of bundles, in which case $h=\mathrm{id}_M$, $\psi=\mathrm{id}_F$ and $\lambda=\mathrm{id}_G$, and we can also take $\alpha=a,\beta=b$ and $\bar{g}_a=e$. Then we get $$t_{\beta\alpha}'=g_\beta t_{\beta\alpha}g_\alpha^{-1}.$$ This is indeed what bundle isomorphisms generally look like, but it doesn't let us make any inferences about what $g_\alpha$'s must be.

Attempt 2: Now let's take some inspiration from Cartan geometry, where the model fiber is actually a homogeneous space $F=G/H=\{Hg\}_{g\in G}$ and the structure group is $H$. This means that there is a transitive left action of $G$ with stabilizers isomorphic to $H$: $G\curvearrowright F$. Then $t_{\beta\alpha}$'s can be assumed to take values in $H$, whereas $g_\alpha$'s can still take values in all of $G$. In this case the last formula obtained for bundle isomorphisms gets replaced with $$t'_{\beta\alpha}g_\alpha t_{\beta\alpha}^{-1}g_\beta^{-1}\in\mathrm{Stab}_G(f),\text{ for all }f\in F$$ but the intersection of all stabilizers is still trivial, so we get $t'_{\beta\alpha}g_\alpha t_{\beta\alpha}^{-1}g_\beta^{-1}=e$. This should somehow imply that $g_\alpha$'s actually commute with some subgroup of $G$ (the group of translations in the vector bundle case), but I don't see how that would happen.

Analysis: my approach so far gives me absolutely no information about what the values of $g$ should actually be, namely elements of a $\mathrm{GL}_k$ subgroup of $G=\mathrm{Aff}_k$. My goal is to come up with a definition of morphisms between bundles with $\mathrm{Aff}$-structures such that it implies that these morphisms must be equivariant with respect to the normal subgroup $\mathbb{R}^k$ of translations, i.e. actual fiberwise linear maps. Perhaps I need to be smart about requiring equivariance of $\psi$, but it's not clear how to do that considering that we need to be able to have different Lie groups acting on the two bundles (e.g. imagine embedding a vector bundle into another vector bundle of another rank, in which case $(G,H)\neq(G',H')$).

Attempt 3 (in response to Ben): every one of these maps $\hat{h}_{\alpha a}$ can be identified with a $G\times G'$-equivariant map $\Phi_{\alpha a}:U_{\alpha a}\times(G\times G')\to C^{\infty}(F,F')$ as follows: $$\Phi_{\alpha a}:\quad (p,g,g')\mapsto \left(f\mapsto g' \hat{h}_{\alpha a}(h(p),g^{-1}f)\right).$$ The action of $G\times G'$ on $C^{\infty}(F,F')$ is by composition $(g,g',\varphi)\mapsto g'\circ\varphi\circ g^{-1}$. I would like to set some restriction on these maps so that I could say that $h$ respects the $G$- and $G'$-structures in some sense. I can do it by analogy with the interpretation of a regular $G$-structure: require all $\Phi_{\alpha a}$'s to take values in a single orbit of the action of $G\times G'$ on $C^\infty(F,F')$. This is a well-defined restriction because transition functions take values in $G\times G'$.

Let see what effect this has in the context of vector bundles. Here $G$ and $G'$ are some $\mathrm{GL}$-groups, say $G=\mathrm{GL}(V)$ and $G'=\mathrm{GL}(V')$. Then to say that a collection of maps $\varphi_p:F\to F'$ belongs to a single orbit of $\mathrm{GL}(V)\times \mathrm{GL}(V')$ acting on $C^\infty(F,F')$ is equivalent to saying that there exist vector space structures on $F,F'$ (isomorphic to $V$ and $V'$ respectively) which make all of these maps linear. This then leads to the standard definition of vector bundles.