This is answered by Noah Olander here: https://arxiv.org/abs/2110.09256