To my mind there are two classes of interesting categorical facts here, loosely speaking "additive" facts and "multiplicative" facts. Some additive facts:
Finite-dimensional vector spaces over $k$ has biproducts, and every object is a finite biproduct of copies of a single object, namely $k$. The categories with this property are precisely the categories of finite rank free modules over a semiring $R$ (the endomorphisms of the single object). These biproduct decompositions encapsulate both the idea that vector spaces have bases and that a choice of bases can be used to write linear maps as matrices.
The single object $k$ above is simple, and so every object is a finite biproduct of simple objects. The categories with this property, in addition to 1, are precisely the categories of finite rank free modules over a division semiring $R$. I believe this is enough to power Gaussian elimination and so forth.
Note that additive facts can't see anything about fields being commutative. The multiplicative facts can:
Finite-dimensional vector spaces over $k$ is symmetric monoidal with respect to tensor product, and is also closed monoidal and has duals (sometimes called compact closed). This observation encapsulates the yoga surrounding tensors of various types (e.g. endomorphisms $V \to V$ correspond to elements of $V \otimes V^{\ast}$), as well as the existence and basic properties (e.g. cyclic symmetry) of the trace.
The single object $k$ above is the tensor unit, and so every object is a finite biproduct of copies of the unit. Also, the monoidal structure is additive in each variable. I believe, but haven't carefully checked, that the (symmetric monoidal) categories with this property, in addition to 1 and 2 above, are precisely the (symmetric monoidal) categories of finite rank free modules over a commutative division semiring ("semifield") $R$. This encapsulates the concrete description of tensor products as a functor in terms of Kronecker products.
There's surprisingly little to say as far as fields being rings as opposed to semirings, though. This mostly becomes relevant when we reduce computing (co)equalizers to computing (co)kernels by subtracting morphisms, as in any abelian category.