I think:

- The category of varieties over $\mathbb Q$ is already very arithmetic.
- One reason that the linearization is considered arithmetic is that so much of the tractable arithmetic information is preserved by linearization, so our arithmetic tools can be used to study it (on the other hand, we have many powerful geometric tools to study structure that is lost by linearization).
- Another reason is that the linearized category of varieties over $\mathbb Q$ is very close to the linearized category of varieties over $\mathbb C$ (closer than the corresponding nonlinearized categories), so even purely geometric questions about the linearization are related to arithmetic.

I will justify these claims and try to explain them.

For the first claim, the category of varieties over $\mathbb Q$ contains the solutions to all problems about existence of rational points on varieties, and many related problems, manifested in the Hom sets from points to varieties. These are certainly classical arithmetical problems, some of the oldest. In fact, homomorphisms from $X$ to $Y$ over $\mathbb Q$ are the same as $\mathbb Q$-points of the Hom-scheme from $X$ to $Y$, so in a sense all, or almost all, the information in the category is Diophantine in nature.

For the second claim, this has to do with the fact that Galois representations on vector spaces are many times easier to understand than Galois actions on nonabelian fundamental groups, let alone the full ensemble of Galois actions that are needed to study the category of varieties over $\mathbb Q$. Note that, while we are making rapid progress in the study of Galois actions on fundamental groups, much of the progress relies on using what we already know about Galois representations, their cohomology, and related objects in increasingly clever ways. By linearization, we lose the structure we don't understand and keep the structure we do understand by using everything from elementary group theory to the Langlands correspondence.

For the third claim, note that if we have two varieties $X$ and $Y$ over $\mathbb Q$, the space $Hom(X,Y)$ in some nice linearization of the category of varieties over $\mathbb C$ will be a finite-dimensional vector space over $\mathbb Q$ (the motives defined using homological and numerical equivalence both have this property). It will admit a Galois action, which because it is an action on a finite-dimensional vector space will factor through a finite group. So it seems we need only a finite amount of information, and relatively simple information at that, to pass from the geometric category to the arithmetic category.

The reason for this is that, according to the Tate conjecture, the geometric category was secretly an arithmetic object all along - the Hom spaces are homomorphisms of Galois representations that are equivariant with respect to the identity component of the Galois group.