Does there exist an operator, $\star$, such that for all full rank matrices $B$ and all $A$ of appropriate dimensions: $$ B(B^\intercal AB)^\star B^\intercal = A^\star, $$ and such that $A^\star=0$ if and only if $A=0$?
Edit: Also, $\star : \operatorname{M}(m,n,\mathbb R) \to \operatorname{M}(n,m,\mathbb R)$.
Edit: If possible, we would also like $\operatorname{rank}(A^\star)=\operatorname{rank}(A)$.
No. We suppose it is defined for $2\times 2$ matrices and we get a contradiction. Unless you drop the requirement on $\operatorname{rank} A^\star$ in which case $A^\star=0$ trivially works.
Let $A=\begin{pmatrix} 1 & 0 \\ 0 & 0\end{pmatrix}$ and let $B=\begin{pmatrix} a & b\\ c & d\end{pmatrix}$ with arbitrary $a,b,c,d\in\mathbb R$ satisfying $ad-bc\neq 0$.
The key observation is that $B^T A B = \begin{pmatrix} a^2 & ab \\ ab & b^2\end{pmatrix}$ does not depend on $c,d$. Therefore $(B^T A B)^\star=B^{-1}A^\star (B^T)^{-1}$ should be independent of $c,d$ as well.
Let $A^\star =\begin{pmatrix} x & y\\ z & w\end{pmatrix}$ for some $x,y,z,w\in\mathbb R$. Then we compute $$B^{-1}A^\star (B^T)^{-1}=\frac 1 {(\operatorname{det} B)^2}\begin{pmatrix} x d^2 - (y+z) cd + w c^2 & -x bd +(yad + zbc) -wac\\ -x bd +(ybc + zad) -wac & x b^2 - (y+z) ab + w a^2 \end{pmatrix}$$.
In case $B=\begin{pmatrix} 1 & 0\\ t & 1\end{pmatrix}$ we get that $B^{-1}A^\star (B^T)^{-1}=\begin{pmatrix} x - (y+z)t + w t^2& \dots\\ \dots & \dots\end{pmatrix}$ is independent of $t$. So $y+z=0$ and $w=0$. Now, in case $B=\begin{pmatrix} 0 & -1\\ 1 & t\end{pmatrix}$ we get $B^{-1}A^\star (B^T)^{-1}=\begin{pmatrix} x t^2 + 0 & \dots\\ \dots & \dots\end{pmatrix}$ from which $x=0$.
This is already a contradiction, because $A^\star=\begin{pmatrix} 0 & y \\ -y & 0\end{pmatrix}$ has either rank 0 or 2. But in case $B=\begin{pmatrix} 1 & 0\\ 0 & t\end{pmatrix}$ we get that $B^{-1}A^\star (B^T)^{-1}=\frac 1 {t^2}\begin{pmatrix} 0 & yt \\ -yt & 0\end{pmatrix}$ is independent of $t$, so $y=0$.
