In order to do geometry, you need to have some kind of global structure which has good local models (the "neighborhoods") and good gluing conditions. In algebraic geometry, the good local models are rings. If you want do geometry with a fibered category, you need gluing conditions (that is, you need your fibered category to be a stack), and you need local models, that is, you need your category to be locally, in some pre-topology, an affine scheme (this is not quite right, but I hope it gives a rough idea). The pre-topology must be such that if $X \to Y$ is a covering, the fact that $Y$ has certain "interesting" local properties implies that $X$ also has them. Étale coverings work very well, of course; smooth coverings also work, not quite as well.
So, you can't do geometry with the stack of coherent sheaves because this does not have good neighborhoods. See also my answer to Qcoh(-) algebraic stack? to see what can wrong.
As to why algebraic stacks are always assumed to be stacks in groupoids, there are several things I could say, but the honest answer is that I don't know the deep reason for this. I know that in practice it suffices, so there is no reason to give up the inversion map, which is quite useful. Just think of how much more you can say about group actions, than about actions of monoids.
Of course, this does not mean that in the future people will not feel the need to extend the theory of algebraic (or topological, or differentiable) stacks to the more general case.
[Edit]: So, why is a geometric stack a stack in groupoids? Well, the first reason is that the inversion map is very useful in proving results. Of course, if we needed to do without it, we would.
The second, more serious, reason, is that, in concrete examples, stacks with non-cartesian maps tend not to admit non-trivial map to spaces. For example, consider the stack $\mathcal M_{1,1}$ of elliptic curves. If we admitted all squares as morphisms, instead of only the cartesian ones, any map from $\mathcal M_{1,1}$ to a space would have to collapse an isogeny classes of curves to a point, and then one can see that it would map everything to a point. So, no moduli space.
As another example, take the stack of vector bundles on a projective variety $X$. There is a map between any two vector bundles, so no open substack could possibly admit a non-trivial map to a space.
Of course, if $F$ is a stack over a site $C$, there is substack $F^*$ with the same objects, whose arrows are the cartesian arrows in $F$; and if $X$ is an object of $C$, or a sheaf on $C$, any cartesian functor $X \to F$ would factor through $F^*$; so you could argue that a chart for $F$ would in fact come from a chart for $F^*$. In all the examples I know, $F^*$ is the right object to consider.
But, once again, none of these reasons is really compelling; for example, if monoid actions became important in geometry, I would bet that soon people would start working with geometric stacks that are not stacks in groupoids.
