I did a little work on this problem in Bielefeld in 1991, based on a suggestion by Waldhausen.

Let $G$ be a finite group. The algebraic $K$-theory $A^G(X)$ of the category of finite retractive $G$-spaces and $G$-maps over and under $X$, with $G$-homotopy equivalences as the weak equivalences, does indeed have a Segal-tom Dieck style factorization in terms of non-equivariant $A$-theory, indexed over the conjugacy classes of subgroups $H$ of $G$, as you describe. This follows from the additivity theorem. However, I was unable to realize this as the $G$-fixed points of a $G$-spectrum $A_G(X)$.

If you naively define the latter as the algebraic $K$-theory of the $G$-category with the same objects as above, but consider all (not necessarily $G$-equivariant) maps under and over $X$ as morphisms, and take the non-equivariant homotopy equivalences as the weak equivalences, then its $G$-fixed part $A_G(X)^G$ is instead the algebraic $K$-theory of the category of finite retractive $G$-spaces and $G$-maps, but with respect to a coarser notion of weak equivalence, namely a $G$-equivarant map whose underlying map is a homotopy equivalence

There was some interest in trying to prove a version of the Segal conjecture for this theory, i.e., to see if
$$
A_G(X)^G \to A_G(X)^{hG}
$$
is an equivalence after suitable completion. This could then contain the original Segal conjecture/Carlson's theorem about
$Q_G(X_+)^G \to Q_G(X_+)^{hG}$
as a retract. I observed that in general there could not be such an equivalence for a version of $A_G(X)$ satisfying the Segal-tom Dieck splitting
$$
A_G(X)^G \simeq \prod_{(H)} A(X^H_{hW_GH})
$$
and the other expected formula
$$
A_G(X)^{hG} \simeq F(BG_+, A(X))
$$
(for $X = X^G$, probably). The argument used your (Goodwillie's) computation of the derivative of $A \colon X \mapsto A(X)$ in terms of $\Sigma^\infty(\Lambda X)_+$, to see that the derivatives of the two right hand sides above were not equivalent. Basically free loop spaces do not commute with Borel constructions. So my conclusion was while the Segal conjecture could hold for one space, it would then mostly fail in a neighborhood of that space.