Let $V$ be a variety over a field $K$.  In the category of varieties (or schemes) over $K$.  I am very interested in studying all the morphisms from $\operatorname{Spec}(K)$ to $X$.  I could hardly be less interested in studying the set of all morphisms from $X$ to $\operatorname{Spec}(K)$: this is, trivially, a single point.

On the other hand, if your variety $V$ is affine -- say $\operatorname{Spec} A$ -- then we are really saying that we prefer to study $K$-algebra maps from $A$ to $K$ (i.e., maps *from* $A$) rather than $K$-algebra maps from $K$ to $A$ (i.e., maps *into* $A$).  This points to a curious feature of your question: it is probably most natural to construe it in terms of categories.  But in this setup, if you just switch to the opposite category, the answer switches around!

Nevertheless I think your question is a real one.  One could just as well ask: why do most categories come with a "natural orientation", i.e., why do we prefer the category to its opposite category?  I think there's something to this question as well.