I think the answer is obvious. If you want to stay in mathematics, quit HTT and go back to the basics of what got you interested in math in the first place. It's easy for young people to be seduced into highly abstract areas just because there is a community of senior mathematicians around them doing it, and I've seen it many times.

Your assumption that you don't have enough time to learn something new is incorrect. It does not take long to learn something different and even publish in that field if your heart is in it. Do another postdoc and take time to explore your own interests, publish and find a new community.