Let $S_i$ be the set of monic monomials $m \in \mathbb{Z}[x_1, \dots, x_k]$ which are divisible by $x_i$ but not $x_i^2$. If I am reading your question correctly, you are looking for $|S_1 \cup \cdots \cup S_k|$.
Note that for $1 \leq i_1 < \dots < i_m \leq k$, the intersection $S_{i_1} \cap \cdots \cap S_{i_m}$ is the set of monomials of degree $n$ divisible by $x_{i_1} \cdots x_{i_m}$ but not by $x^2_{i_1} \cdots x^2_{i_m}$. If $m < n$ and $m < k$, then there is a bijection between $S_{i_1} \cap \cdots \cap S_{i_m}$ and the set of monic monomials of degree $n-m$ in $\mathbb{Z}[x_1, \dots, x_{k-m}]$. (If $m = n \leq k$, then the intersection has one element, $x_{i_1} \cdots x_{i_m}$. In any other case, the intersection is empty.) Hence, for $1 \leq i_1 < \cdots < i_m \leq k$, $$|S_{i_1} \cap \cdots \cap S_{i_m}| = \begin{cases} \left(\matrix{n + k - 2m -1 \cr k - m - 1}\right), & \text{if $m < n$ and $m < k$} \cr 1, & \text{if $m = n \leq k$} \cr 0, & \text{otherwise.}\end{cases}$$
So, by the principle of inclusion-exclusion, $$|S_1 \cup \cdots \cup S_k| = \sum_{m =1}^{\min(n, k)-1} (-1)^{m-1} \left(\matrix{k \cr m}\right)\left(\matrix{n + k - 2m - 1 \cr k - m - 1}\right) + (-1)^n -1)^{n-1} \left(\matrix{k\cr n}\right)a,$$ where $$a = \begin{cases} 1, & \text{if $k \geq n$} \cr 0, & \text{otherwise.}\end{cases}$$
