Certainly it is possible to "jump ship" and, as the comments have already pointed out, a few years in a physics PhD program (where you published math papers) will not damage your credibility. In fact, I'd encourage you to **market it as a strength.** When you apply to math grad school, the fact is that you already have math publications (most other applicants don't). You already know how to do research. You already know how to pass qualifying exams and give talks. You are a lot stronger than the other applicants. In terms of letters of recommendation, I'd pick your favorite professor and ask that person. I'm sure they care about your well-being in life and would not want you to do a career you hate. This professor can speak to your strengths as a student, your strengths as a researcher, your passion for math, and how you got along with people in the department. That's plenty enough to get into a PhD program in math. The letter doesn't have to be from your physics PhD advisor.

I'd also market your time in physics as a strength when you do eventually apply for jobs after finishing the math PhD. I've been on about 10 hiring committees and can remember several cases where people had jumped ship, including a few where the person had actually already finished their first PhD then jumped ship and gotten a second PhD. We were really impressed with these candidates, and I remember one case where the person got a job offer from another university before we could even schedule an on-campus interview. 

So, instead of saying "I hated physics and quit" you want to say something to the effect of "I have experience doing research in both mathematics and physics. This enables me to do **interdisciplinary research**, and to **create collaborations** with professors both in and outside of your department. It also **informs my teaching**, as I can better connect with math students who favor a physical intuition. And, it **informs my advising** of students: how to make decisions based on their passion, how to trust their instincts, and how to be resilient."

Now, I want to be clear that I'm at a liberal arts college, where teaching and advising are super important, and where we like candidates with a breadth of knowledge and the possibility to do interdisciplinary research. Even if your math research has nothing to do with physics, it's a benefit that you have such a deep knowledge of physics. It could help, for example, in teaching math courses aimed at students trying to do engineering or physics grad school (like, div, grad, curl for physicists, heat equation, etc). These days even big research universities care about your teaching, so I don't think the physics time would hurt you.

One last point. You've already sunk six years into the physics PhD. All else being equal, it would be better to have finished it before starting the math PhD (which, by the way, you can probably do a lot faster than six years with all you've learned about research) rather than to leave with no degree. If you think you can finish it within a year or so, I'd encourage you to do that, even if you plan to jump ship.