In Jack's example the fiber is not scheme-theoretically $\mathbb A^1$. You can get a counterexample by taking $Y$ to be a nodal curve, $Y'$ its the normalization, with one of the two points in the inverse image of the node removed, and $X = Y' \times \mathbb A^1$.
If we assume that the map is smooth, this becomes quite subtle. It is false in positive characteristic. Let $k$ be a field of characteristic $p > 0$. Take $Y = \mathbb A^1 = \mathop{\rm Spec}k[t]$, $Y' = \mathop{\rm Spec} k[t,x]/(x^p - t)$. Of course $Y' \simeq \mathbb A^1$, but the natural map $Y' \to Y$ is an inseparable homemorphism. Now embed $Y'$ in $\mathbb P^1 \times Y$ over $Y$, and take $X$ to be the difference $\mathbb (P^1 \times Y) \smallsetminus Y'$$(\mathbb P^1 \times Y) \smallsetminus Y'$.
On the other hand, it is not so hard to show that in characteristic 0 the answer is positive for $n = 1$ (if $Y$ is reduced), and I believe it is known to be true $n = 2$. The general case seems estremely hard.
I am afraid that Sasha'a argument does not work; if the fiber does not have a vector spaces structure, there is not reason that choosing points gives you a trivialization.
[Edit] The question has been updated with "what if one assumes vector space structure on the fibres?"?
Well, $\mathbb A^n$ can always be given a vector space structure. In my first example, the fibers are canonically isomorphic to $\mathbb A^1$, so they have a natural vector space structure.
However, if the map $X \to Y$ is smooth, and the vector space stucture is allowed to "vary algebraically" that is, if the zero section $Y \to X$ is a regular function, the addition gives a regular function $X \times_Y X \to X$, and scalar multiplication gives a regular function $\mathbb A^1 \times X \to X$, then $X$ is in fact a vector bundle. The proof uses some machinery: one uses smoothness to construct bases locally in the étale topology, showing that $X$ is étale locally trivial over $Y$, and descent theory to show that in fact $X$ is Zariski locally trivial.