Taking Leray Jenkins' hint that integrality is preserved, I think I've figured it out. Let $F$ be a totally real field of degree $[F:\mathbb{Q}] = d$, Galois over $\mathbb{Q}$. Non-parallel weights correspond not to characters $\operatorname{Res}_{\mathcal{O}_F/\mathbb{Z}}\mathbb{G}_m \to \mathbb{G}_m$, but to characters $\operatorname{Res}_{\mathcal{O}_F/\mathbb{Z}}\mathbb{G}_m \to \operatorname{Res}_{\mathcal{O}_F/\mathbb{Z}}\mathbb{G}_m$. Parallel weights simply have their image in the subgroup $\mathbb{G}_m \subset \operatorname{Res}_{\mathcal{O}_F/\mathbb{Z}}\mathbb{G}_m$.
For non-parallel weights, we fix a base embedding $\sigma_0 \colon F \to \mathbb{R}$. This identifies $\Sigma$, the set of real embeddings of $F$, with $\operatorname{Gal}(F/\mathbb{Q})$ by $g \leftrightarrow \sigma_0 \circ g^{-1}$. Over $\mathbb{R}$, we consider $(k_\sigma)_{\sigma \in \Sigma}$, corresponding to the character $\chi \colon \lambda \mapsto \prod_{\sigma} \sigma(\lambda)^{k_\sigma}$. We build $\chi_{\sigma_0} \colon \operatorname{Res}_{\mathcal{O}_F/\mathbb{Z}}\mathbb{G}_m \to \operatorname{Res}_{\mathcal{O}_F/\mathbb{Z}}\mathbb{G}_m$ so that $\sigma_0(\chi_{\sigma_0}) = \chi$. Define $g_0^\sigma$ so that $\sigma(\lambda) = \sigma_0(g_0^\sigma (\lambda))$. Then we fulfill that requirement with
$$ \chi_{\sigma_0}(\lambda) = \prod_{\sigma \in \Sigma} (g_0^\sigma \cdot \lambda)^{k_\sigma}. $$
We have the character $\chi_{\sigma_0} \colon \operatorname{Res}_{\mathcal{O}_F/\mathbb{Z}}\mathbb{G}_m \to \operatorname{Res}_{\mathcal{O}_F/\mathbb{Z}}\mathbb{G}_m$, defined over $\mathbb{Z}$. It depends on the choice of $\sigma_0$, and the weight $(k_{g^{-1} \cdot \sigma})_\sigma$, along with the choice $g \cdot \sigma_0$ as our base (possibly $g^{-1}\cdot \sigma_0$) will give the same integral character, and so the same modular forms; $g$ or $g^{-1}$ should give an algebraic isomorphism between the relevant spaces of modular forms over $\mathbb{R}$, meaning this shouldn't matter.
This gives us an algebraic character for our weight ($\chi_{\sigma_0}$ is defined over $\mathbb{Z}$), and landing the characters in $\operatorname{Res}_{\mathcal{O}_F/\mathbb{Z}}\mathbb{G}_m$ rather than just $\mathbb{G}_m$ is okay since the values of Hilbert modular forms defined over $R$ are allowed to be in $\mathcal{O}_F \otimes_\mathbb{Z} R$ anyway. One can check that this reduces to the character $(Norm_{\mathcal{O}_F/\mathbb{Z}})^k$ for the case of parallel weight $k$.