I've heard from others about the WO($\kappa$) as a counterpart of AC($\kappa$), but I cannot find a suitable way to express it in ZF since "every set of cardnality $\kappa$ can be well-ordered" is meaningless. Does WO($\kappa$) really exist?
I've found one. "Every set in $V_\kappa$ can be well-ordered." Is this right?
The counterpart you may intend to use is that every cardinal is comparable with $\kappa$. This means that every infinite set which is not smaller than $\kappa$ has a subset of cardinality $\kappa$.
This is the restriction of the well-ordering principle to $\kappa$, whereas $\sf AC_\kappa$ is the restriction of the axiom of choice to families of size at most $\kappa$.
It should be noted that $\sf WO_\kappa$ does not imply $\sf AC_\kappa$, and the other way around is true only if $\sf WO_{<\kappa}$ holds and $\sf AC_{\operatorname{cf}(\kappa)}$ holds for limit cardinals.
For more on that, see Jech The Axiom of Choice, Chapter 8.
Your idea, while interesting has a problem that we can't really control the cardinality of the sets in $V_\kappa$. We would like something that we can somewhat control, which is comparability of cardinals. In a sense the well-ordering principle is just a prelude to the trichotomy theorem (every two cardinals are comparable), so this is what we want to limit.
In the same breath, I should mention that my philosophy is that $\sf DC_\kappa$ is the true restriction of $\sf AC$ to $\kappa$, being stronger than the two principles above, and corresponding to Zorn's lemma (or maybe Hausdorff's maximality principle).
