Although this is rather a comment (or at best a partial answer) than an (honest) answer, I don’t write it as such. But it’ll contain some thoughts about my question and so it seems appropriate to write it as an answer (at least to me…).
@Jason Starr: Of course you’re completely right. Sorry, I was probably too sloppy by only writing “Riemann surface” than “compact (connected) Riemann surface”. Maybe I hoped that it would be clear from the context.
In any case, your (and Jack’s) comment sparked my interest in GAGA and how it really applies.
@YangMills: Today I was able to read some passages of Kobayashi’s “Differential geometry of complex vector bundles” and it clarified some of my confusions. Thanks!
So here are my thoughts about my question:
Let $X$ be a compact Riemann surface and denote by $O_X$ the sheaf of holomorphic functions on $X$. Then there is a bijection between holomorphic vector bundles of rank $r$ over $X$ and locally free $O_X$-modules of rank $r$. But what are the morphisms on each side?
The morphisms of locally free $O_X$-modules (of rank $r$) are simply sheaf homormorphisms.
Maybe all authors (at least for example Huybrechts or Kobayashi) define a holomorphic vector bundle morphism $\varphi:V\to W$ (where $V$ and $W$ are holomorphic vector bundles over $X$) as a holomorphic map $\varphi:V\to W$ such that the restriction of $\varphi|x$ to any fiber $V(x)$ maps complex linearly to $W(x)$ AND the rank of $\varphi|x$ is independent of $x\in X$ (call this definition “definition B”).
In my question I used an alternative definition (“definition A”) of a holomorphic vector bundle morphism, i.e. I didn’t require the rank to be constant. If one uses the definition B instead, the image of a holomorphic vector bundle morphism $\varphi: V\to V$ is indeed a holomorphic subbundle of $V$ and one can proof that $H^0(End(V))=H^0(Iso(V))$ just as in the “sheaf case”.
But with definition A one can prove this fact as well by using sheaf theory because we can define stability (resp. slope $\mu(P)$) for every torsion-free coherent sheaf $P$ over $X$. Without going into detail I shall just mention that stability for $P$ means that $$\mu(P')<\mu(P)$$ for every coherent subsheaf $P'$ with $0 < rank(P') < rank(P)$.
Now every holomorphic vector bundle homomorphism $\varphi:V\to V$ (w.r.t definition A) induces a sheaf homorphism $\phi:S\to S$ of the underlying locally free $O_X$-sheaf $S$ and vice versa. (This fails in definition B.) The image of $S$ under $\phi$ is in general no longer locally free (claim: this is precisely the case when $\varphi$ has constant rank). But we can construct the (coherent) image sheaf $im S$ of $S$ so that the slope of $im S$ is defined. Then we can proceed as I indicated in my question (cf. Kobayashi’s “Differential geometry of complex vector bundles”) and we see that $\phi$ and hence $\varphi$ is an isomorphism.
Even if this seems quite plausible I’m still not really satisfied because the above stability condition seems to be much stronger than the one for a holomorphic vector bundle $V$ since one considers not only locally free subsheaves (i.e. holomorphic subbundles of $V$) but also coherent subsheaves of the corresponding locally free sheaf.
So my feeling is that if one uses definition A (i.e. omitting constant rank) then $H^0(End(V))=H^0(Iso(V))$ is in general only true under this stronger stability condition.
(But it’s also possible that this is complete non-sense…)