There are two candidates for a category whose objects are vector bundles on $X$.  Candidate (1) is the one you and Ravi Vakil's notes describe: the morphisms are all maps of $\mathcal{O}_X$-modules.  As you observe, this fails to have the obvious kernels and cokernels, and since every coherent sheaf is locally a quotient of free $\mathcal{O}_X$-modules, the actual ones have to coincide with the obvious ones (which are the co/kernels in the category of coherent sheaves).

Candidate (2) has the morphisms being only those of *constant rank*; equivalently, those for which the sheaf quotient is a vector bundle.  This makes the sheaf kernel a vector bundle as well, but this candidate has several problems identified in the comments and is as a result neither abelian nor a category.