There is a reference in my comment above (Lemma 2.1.5 in arxiv.org/pdf/1811.02057.pdf). Let me repeat the argument from there in the answer here.
Let $X$ be a $\pi$-finite space and $q:X\to pt$ be the projection to the point. We assume that the norm map $Nm_q : q_! \to q_*$ is an isomorphism when evaluated at "trivial modules", which are just functors of the form $q^*A$. We want to prove that it is an isomorphism at every object.
For an object $A$, the zigzag indentity gives us a retract diagram $q_*A \to q_*q^*q_*A \to q_*A$. Now, the norm map $Nm: q_! q^*q_*A \to q_*q^*q_*A$ is an isomorphism by the assumption, and the naturality of the norm map gives us that the composition $q_*A\to q_*q^*q_*A \stackrel{Nm^{-1}}{\to} q_!q^*q_*A \to q_!A \stackrel{Nm}{\to}q_*A$ is the identity (where the unlabeled arrows are the usual units and counits). Hence, $Nm: q_! A \to q_*A$ is invertible from the left. A similar dual argument with the Zigzag identity for $q_!$ and $q^*$ gives us that it is right invertible, hence an isomorphism.
The intuition here is that taking homotopy orbits/fixed points twice just enlarge the object so this way we reduce from general module to a trivial one.
