There are two candidates for a category whose objects are vector bundles on $X$. Candidate (1) is the one you and Ravi Vakil's notes describe: the morphisms are all maps of $\mathcal{O}_X$-modules. As you observe, this fails to have the obvious kernels and cokernels, and since every coherent sheaf is locally a quotient of free $\mathcal{O}_X$-modules, the actual ones have to coincide with the obvious ones (which are the co/kernels in the category of coherent sheaves).
Candidate (2) has the morphisms being only those of constant rank; equivalently, those for which the sheaf quotient is a vector bundle. This makes the sheaf kernel a vector bundle as well, and as a result, this definition of $\mathrm{Vect}(X)$ is an abelian category with the obvious co/kernels.