I don't know about jets and you already got an answer regarding the bundle of automorphisms, anyway if you want a $GL_{n,Y}$-torsor over $Y$ that gives you back your original vector bundle $Z=V(\mathcal{E})=Spec_Y Sym(\mathcal{E})\to Y$ when you apply the associate bundle construction with $\mathcal{O}_Y^n$, you should take the bundle of local frames of $Z$, that is $P=\underline{Isom}_Y(\mathbb{A}^n_Y,Z)\to Y$, where $\underline{Isom}$ is the scheme representing the sheaf of isomorphisms. This is a $GL_{n,Y}$-torsor over $Y$ by the action of $GL_{n,Y}$ on $\mathbb{A}^n_Y$, and if you want a sheaf $\mathcal{F}$ of $\mathcal{O}_Y$-algebras such that $P=Spec_Y Sym(\mathcal{F})$, P=Spec_Y(\mathcal{F})$, it seems reasonable (but i didn't really check) that you can take$\mathcal{F}=\underline{Isom}_{\mathcal{O}_Y-\text{mod}}(\mathcal{E},\mathcal{O}_Y^n)$.\mathcal{F}=\underline{Isom}_{\mathcal{O}_Y-\text{alg}}(Sym(\mathcal{E}),\mathcal{O}_Y^n[x_1,..,x_n])$.
I don't know about jets and you already got an answer regarding the bundle of automorphisms, anyway if you want a $GL_{n,Y}$-torsor over $Y$ that gives you back your original vector bundle $Z=V(\mathcal{E})=Spec_Y Sym(\mathcal{E})\to Y$ when you apply the associate bundle construction with $\mathcal{O}_Y^n$, you should take the bundle of local frames of $Z$, that is $P=\underline{Isom}_Y(\mathbb{A}^n_Y,Z)\to Y$, where $\underline{Isom}$ is the scheme representing the sheaf of isomorphisms. This is a $GL_{n,Y}$-torsor over $Y$ by the action of $GL_{n,Y}$ on $\mathbb{A}^n_Y$, and if you want a sheaf $\mathcal{F}$ of $\mathcal{O}_Y$-algebras such that $P=Spec_Y Sym(\mathcal{F})$, it seems reasonable (but i didn't really check) that you can take $\mathcal{F}=\underline{Isom}_{\mathcal{O}_Y-\text{mod}}(\mathcal{E},\mathcal{O}_Y^n)$.