$\newcommand{\To}{\longrightarrow}$$\newcommand{\dash}{\text{-}}$$\newcommand{\sSet}{\mathrm{sSet}}$$\newcommand{\ZZ}{\mathbb{Z}}$For simplicity (and generality), let me denote by $G$ the group $(\ZZ/p)^\times$, and by $X$ the simplicial set $B(\ZZ/p)$. Then $X$ is a pointed $G$-simplicial set. I will prove that $$ X/G \simeq (X\times_G EG)/BG \simeq B(\ZZ/p \rtimes G)/BG $$
As observed in the question, the action of $G$ on $X$ is free on every simplex which is not in the basepoint of $X$. This implies that the inclusion of the basepoint $1\to X$ is a cofibration in the projective model structure on $G$-simplicial sets. In other words, $X$ is a cofibrant object in the projective model structure on pointed $G$-simplicial sets.
Now observe that quotienting out by the action of $G$ is a left Quillen adjoint from pointed $G$-simplicial sets to pointed simplicial sets. I will denote this left Quillen adjoint by $F$: $$ F : 1\downarrow G\dash\sSet \To 1\downarrow\sSet $$
Since $X$ is a cofibrant object in the domain of $F$, the quotient $X/G = F(X)$ is weakly equivalent to $LF(X)$, the left derived functor of $F$ applied to $X$.Finally, the left derived functor of $F$ applied to $X$ is weakly equivalent to the quotient of the Borel construction of $X$ by its subspace $BG$: $$ LF(X) \simeq (X\times_G EG)/BG \rlap{\qquad\qquad\text{(#)}} $$ where $BG$ includes into the Borel construction via the basepoint of $X$: $$ BG = EG/G = 1\times_G EG \To X\times_G EG $$
Proof of (#): Consider the sequence of left Quillen adjoints: $$ EG \downarrow G\dash\sSet \overset{T}{\To} 1 \downarrow G\dash\sSet \overset{F}{\To} 1\downarrow\sSet $$ where the first functor $T$ is left adjoint to pre-composing with $EG\to 1$, i.e. it quotients out by $EG$. Since $EG\to 1$ is a weak equivalence of $G$-simplicial sets, and $G$-simplicial sets form a left proper model category, the first functor $T$ is actually a Quillen equivalence. So $X \simeq LT(X)$, where $X$ is seen as an object under $EG$ via $EG\to 1\to X$, and $LT$ is the left derived functor of $T$. Consequently: $$ LF(X) \simeq LF\circ LT(X) \simeq L(F\circ T)(X) \simeq (X\times_G EG)/BG $$ where the last weak equivalence is obtained by expressing $F\circ T$ as the composite of the left Quillen adjoints $$ EG\downarrow G\dash\sSet \overset{(-)/G}{\To} BG\downarrow\sSet \overset{(-)/BG}{\To} \sSet $$ End of proof.
As observed in the comments, the Borel construction $X\times_G EG = B(\ZZ/p)\times_G EG$ is actually equivalent to $B(\ZZ/p \rtimes G)$, the classifying space of the semi-direct product. In conclusion: $$ X/G \simeq B(\ZZ/p \rtimes G)/BG $$