Let $V$ be a variety over a field $K$. In the category of varieties (or schemes) over $K$. I am very interested in studying all the morphisms from $\operatorname{Spec}(K)$ to $X$. I could hardly be less interested in studying the set of all morphisms from $X$ to $\operatorname{Spec}(K)$: this is, trivially, a single point.

On the other hand, if your variety $V$ is affine -- say $\operatorname{Spec} A$ -- then we are really saying that we prefer to study $K$-algebra maps from $A$ to $K$ (i.e., maps from $A$) rather than $K$-algebra maps from $K$ to $A$ (i.e., maps into $A$). This points to a curious feature of your question: it is probably most natural to construe it in terms of categories. But in this setup, if you just switch to the opposite category, the answer switches around!

Nevertheless I think your question is a real one. One could just as well ask: why do most categories come with a "natural orientation", i.e., why do we prefer the category to its opposite category? I think there's something to this question as well.