3 added 4 characters in body

Let $S_i$ be the set of monic monomials $m \in \mathbb{Z}[x_1, \dots, x_k]$ which are divisible by $x_i$ but not $x_i^2$. If I am reading your question correctly, you are looking for $|S_1 \cup \cdots \cup S_k|$.

Note that for $1 \leq i_1 < \dots < i_m \leq k$, the intersection $S_{i_1} \cap \cdots \cap S_{i_m}$ is the set of monomials of degree $n$ divisible by $x_{i_1} \cdots x_{i_m}$ but not by $x^2_{i_1} \cdots x^2_{i_m}$. If $m < n$ and $m < k$, then there is a bijection between $S_{i_1} \cap \cdots \cap S_{i_m}$ and the set of monic monomials of degree $n-m$ in $\mathbb{Z}[x_1, \dots, x_{k-m}]$. (If $m = n \leq k$, then the intersection has one element, $x_{i_1} \cdots x_{i_m}$. In any other case, the intersection is empty.) Hence, for $1 \leq i_1 < \cdots < i_m \leq k$, $$|S_{i_1} \cap \cdots \cap S_{i_m}| = \begin{cases} \left(\matrix{n + k - 2m -1 \cr k - m - 1}\right), & \text{if m < n and m < k} \cr 1, & \text{if m = n \leq k} \cr 0, & \text{otherwise.}\end{cases}$$

So, by the principle of inclusion-exclusion, $$|S_1 \cup \cdots \cup S_k| = \sum_{m =1}^{\min(n, k)-1} (-1)^{m-1} \left(\matrix{k \cr m}\right)\left(\matrix{n + k - 2m - 1 \cr k - m - 1}\right) + (-1)^n -1)^{n-1} \left(\matrix{k\cr n}\right)a,$$ where $$a = \begin{cases} 1, & \text{if k \geq n} \cr 0, & \text{otherwise.}\end{cases}$$

2 added 28 characters in body

Let $S_i$ be the set of monic monomials $m \in \mathbb{Z}[x_1, \dots, x_k]$ which are divisible by $x_i$ but not $x_i^2$. If I am reading your question correctly, you are looking for $|S_1 \cup \cdots \cup S_k|$.

Note that for $1 \leq i_1 < \dots < i_m \leq k$, the intersection $S_{i_1} \cap \cdots \cap S_{i_m}$ is the set of monomials of degree $n$ divisible by $x_{i_1} \cdots x_{i_m}$ but not by $x^2_{i_1} \cdots x^2_{i_m}$. If $m < n$ and $m < k$, then there is a bijection between $S_{i_1} \cap \cdots \cap S_{i_m}$ and the set of monic monomials of degree $n-m$ in $\mathbb{Z}[x_1, \dots, x_{k-m}]$. (If $m = n \leq k$, then the intersection has one element, $x_{i_1} \cdots x_{i_m}$. In any other case, the intersection is empty.) Hence, for $1 \leq i_1 < \cdots < i_m \leq k$, $$|S_{i_1} \cap \cdots \cap S_{i_m}| = \begin{cases} \left(\matrix{n + k - 2m -1 \cr k - m - 1}\right), & \text{if m < n and m < k} \cr 1, & \text{if m = n \leq k} \cr 0, & \text{otherwise.}\end{cases}$$

So, by the principle of inclusion-exclusion, $$|S_1 \cup \cdots \cup S_k| = \sum_{m =1}^{\min(n, k)-1} (-1)^{m-1} \left(\matrix{k \cr m}\right)\left(\matrix{n + k - 2m - 1 \cr k - m - 1}\right) + (-1)^n a,$$ \left(\matrix{k\cr n}\right)a,$$where$$a = \begin{cases} 1, & \text{if $k \geq n$} \cr 0, & \text{otherwise.}\end{cases}$$1 Let S_i be the set of monic monomials m \in \mathbb{Z}[x_1, \dots, x_k] which are divisible by x_i but not x_i^2. If I am reading your question correctly, you are looking for |S_1 \cup \cdots \cup S_k|. Note that for 1 \leq i_1 < \dots < i_m \leq k, the intersection S_{i_1} \cap \cdots \cap S_{i_m} is the set of monomials of degree n divisible by x_{i_1} \cdots x_{i_m} but not by x^2_{i_1} \cdots x^2_{i_m}. If m < n and m < k, then there is a bijection between S_{i_1} \cap \cdots \cap S_{i_m} and the set of monic monomials of degree n-m in \mathbb{Z}[x_1, \dots, x_{k-m}]. (If m = n \leq k, then the intersection has one element, x_{i_1} \cdots x_{i_m}. In any other case, the intersection is empty.) Hence, for 1 \leq i_1 < \cdots < i_m \leq k,$$|S_{i_1} \cap \cdots \cap S_{i_m}| = \begin{cases} \left(\matrix{n + k - 2m -1 \cr k - m - 1}\right), & \text{if $m < n$ and $m < k$} \cr 1, & \text{if $m = n \leq k$} \cr 0, & \text{otherwise.}\end{cases}$$So, by the principle of inclusion-exclusion,$$|S_1 \cup \cdots \cup S_k| = \sum_{m =1}^{\min(n, k)-1} (-1)^{m-1} \left(\matrix{k \cr m}\right)\left(\matrix{n + k - 2m - 1 \cr k - m - 1}\right) + (-1)^n a,$$where$$a = \begin{cases} 1, & \text{if $k \geq n$} \cr 0, & \text{otherwise.}\end{cases}