$\newcommand{\To}{\longrightarrow}$<!--
-->$\newcommand{\dash}{\text{-}}$<!--
-->$\newcommand{\sSet}{\mathrm{sSet}}$<!--
-->$\newcommand{\ZZ}{\mathbb{Z}}$<!--
-->For simplicity (and generality), let me denote by $G$ the group $(\ZZ/p)^\times$, and by $X$ the simplicial set $B(\ZZ/p)$. Then $X$ is a pointed $G$-simplicial set. I will prove that
$$ X/G \simeq (X\times_G EG)/BG \simeq B(\ZZ/p \rtimes G)/BG $$

As observed in the question, the action of $G$ on $X$ is free on every simplex which is not in the basepoint of $X$. This implies that the inclusion of the basepoint $1\to X$ is a cofibration in the projective model structure on $G$-simplicial sets. In other words, $X$ is a cofibrant object in the projective model structure on pointed $G$-simplicial sets.

Now observe that quotienting out by the action of $G$ is a left Quillen adjoint from pointed $G$-simplicial sets to pointed simplicial sets. I will denote this left Quillen adjoint by $F$:
$$ F : 1\downarrow G\dash\sSet \To 1\downarrow\sSet $$
<!--It is a left Quillen adjoint given that its right adjoint preserves fibrations and acyclic fibrations, by definition of the projective model structure. -->Since $X$ is a cofibrant object in the domain of $F$, the quotient $X/G = F(X)$ is weakly equivalent to $LF(X)$, the left derived functor of $F$ applied to $X$.

Finally, the left derived functor of $F$ applied to $X$ is weakly equivalent to the quotient of the Borel construction of $X$ by its subspace $BG$:
$$ LF(X) \simeq (X\times_G EG)/BG  \rlap{\qquad\qquad\text{(#)}} $$
where $BG$ includes into the Borel construction via the basepoint of $X$:
$$ BG = EG/G = 1\times_G EG \To X\times_G EG $$

**Proof of (#):** Consider the sequence of left Quillen adjoints:
$$ EG \downarrow G\dash\sSet \overset{T}{\To} 1 \downarrow G\dash\sSet \overset{F}{\To} 1\downarrow\sSet $$
where the first functor $T$ is left adjoint to pre-composing with $EG\to 1$, i.e. $T$ quotients out by $EG$. Since $EG\to 1$ is a weak equivalence of $G$-simplicial sets, and $G$-simplicial sets form a left proper model category, the first functor $T$ is actually a Quillen equivalence. So $X \simeq LT(X)$, where $X$ is seen as an object under $EG$ via $EG\to 1\to X$, and $LT$ is the left derived functor of $T$. Consequently:
$$ LF(X) \simeq LF\circ LT(X) \simeq L(F\circ T)(X) \simeq (X\times_G EG)/BG $$
The last weak equivalence is obtained by expressing $F\circ T$ as the composite of the left Quillen adjoints
$$ EG\downarrow G\dash\sSet \overset{(-)/G}{\To} BG\downarrow\sSet \overset{(-)/BG}{\To} 1\downarrow\sSet $$
and noting that since $EG$ is cofibrant, the left derived functor of $(-)/G$ is weakly equivalent to the Borel construction.
**End of proof.**

As observed in the comments, the Borel construction $X\times_G EG = B(\ZZ/p)\times_G EG$ is actually equivalent to $B(\ZZ/p \rtimes G)$, the classifying space of the semi-direct product. In conclusion:
$$ X/G \simeq B(\ZZ/p \rtimes G)/BG $$