The question as originally posed was to compute the integral of the trivariate function
\begin{equation}
4 \mu ^{3 \beta +1} \rho_{11}^{3 \beta +1} \left(-\rho_{11}-\rho_{22}+1\right){}^{3 \beta +1}
\rho_{22}^{3 \beta +1} \left(\mu ^2 \rho_{22}+\rho_{11}\right){}^{-3 \beta -2}.
\end{equation}
Matt F. immediately suggested “streamlining” it to the form (transforming $\beta$ into $\frac{b-1}{3}$)
\begin{equation}
\int_{p=0}^1 \int_{q=0}^{1-p} (\mu p q (1-p-q))^b (\mu^2 q +p)^{-b-1} dq dp,
\end{equation}
which has been employed here since.

Subsequent computations, then, revealed that the two-fold integrals for $b=1,2,\ldots$ took the form
\begin{equation}
v(b,\mu) + w(b,\mu) \log(\mu).
\end{equation}
In a comment to the question, Rubey was able to express the coefficient $w(b,\mu)$ of the $\log{\mu}$ term as
\begin{equation}
\frac{b \left(\mu ^2-1\right)^{-2 b-1} \sum _{k=0}^b \mu ^{2 k} \binom{b}{k}^2}{4 \left(4
b^2-1\right) \binom{2 (b-1)}{b-1}}
\end{equation}
(note, in particular, the summation of a squared binomial).
This required an additional factor of $4 \mu^b$, as well as a notational correction of $m$ to $\mu$ (as Rubey agreed in a subsequent comment).

Mathematica, interestingly, converted the Rubey expression (performing the indicated summation) to the hypergeometric-based formula,
\begin{equation}
w(b,\mu)=\frac{\sqrt{\pi } 4^{-b} \mu^b \left(\mu^2-1\right)^{-2 b-1} \Gamma (b+1) \,
_2F_1\left(-b,-b;1;\mu^2\right)}{\Gamma \left(b+\frac{3}{2}\right)}.
\end{equation}

In his later answer to the question, Rubey showed that $w(b,\mu)$ and $v(b,\mu)$ were both given by the same second order recurrence, but with different initial conditions. Making use of these results we have found that
\begin{equation}
v(b,\mu)= -\frac{\sqrt{\pi } 4^{-b} b! \mu ^b \left(\mu ^2-1\right)^{-2 b} \sum _{k=1}^b \mu ^{2
k-2} \sum _{i=0}^{k-1} \binom{b}{i}^2 (\psi ^{(0)}(b-i+1)-\psi ^{(0)}(i+1))}{\Gamma
\left(b+\frac{3}{2}\right)},
\end{equation}
where $\psi$ denotes the polygamma function. (The process of deriving this formula, in part, was the subject of my posting https://math.stackexchange.com/questions/3115582/do-these-polynomials-with-harmonic-number-related-coefficients-lie-in-some-parti .)

The $v(b,\mu)$ and $w(b,\mu)$ formulas have been explicitly found to yield the two-fold integration results for $b=1,2,\ldots,11$.

Most interestingly now, we observe that unlike $w(b,\mu)$, this formula for $v(b,\mu)$ does not appear readily to have an equivalent hypergeometric expression (despite what seem like simpler initial conditions for $v(b,\mu)$ in the second order recurrences). Both formulas involve the summation of a squared binomial, but the summand now has an added polygamma-based factor.

This phenomenon (failure to convert the summation) would appear to be a complicating factor in the
underlying quantum-information-theoretic question we have been hoping to address.
This involves finding “separability” functions $f(b,\mu)$ which when multiplied by the normalization (by $\frac{\Gamma (b)^4}{\Gamma (2 b+2)}$) of $v(b,\mu) +w(b,\mu) \log{\mu}$, and integrated over $\mu \in [0,1]$ would yield certain target (separability probability) values, given by
\begin{equation}
1-\sqrt{\pi } 2^{-\frac{3 b}{2}-1} \Gamma \left(\frac{5 b}{12}+\frac{47}{24}\right)
\Gamma \left(\frac{b}{2}+1\right) \Gamma \left(\frac{5 b}{6}+\frac{7}{6}\right) \Gamma
\left(\frac{2 (b+2)}{3}\right) \, _6\tilde{F}_5\left(1,\frac{1}{6} (2
b+7),\frac{1}{12} (5 b+7),\frac{1}{12} (5 b+13),\frac{1}{24} (10
b+47),\frac{b+2}{2};\frac{b+11}{6},\frac{1}{24} (10 b+23),\frac{5
b}{12}+\frac{4}{3},\frac{1}{12} (5 b+22),\frac{2 (b+2)}{3};1\right)
\end{equation}
(where the regularized hypergeometric function is indicated). For $b=4, 7,10$, this formula gives $\frac{29}{64}$, $\frac{8}{33}$ and $\frac{26}{323}$, the two-rebit, two-qubit and two-"quaterbit" Hilbert-Schmidt separability probabilities, respectively. https://arxiv.org/abs/1701.01973 (Any advice pertaining to this matter would be appreciated.)

Assembling the two functions of interest in the question posed, we can assert that
\begin{equation}
v(b,\mu) + w(b,\mu) \log(\mu)=
\end{equation}
\begin{equation}
\frac{1}{\Gamma \left(b+\frac{3}{2}\right)} \sqrt{\pi } 4^{-b} \mu^b \left(\mu^2-1\right)^{-2 b-1} \Gamma (b+1) \left(\log (\mu) \sum _{k=0}^b \mu ^{2 k} \binom{b}{k}^2-\left(\mu^2-1\right) \sum _{k=1}^b \mu^{2 k-2} \sum
_{i=0}^{k-1} \binom{b}{i}^2 (\psi ^{(0)}(b-i+1)-\psi ^{(0)}(i+1))\right),
\end{equation}
where, as previously noted,
\begin{equation}
\sum _{k=0}^b \mu ^{2 k} \binom{b}{k}^2=\, _2F_1\left(-b,-b;1;\mu ^2\right).
\end{equation}
(Perhaps the "master equation" just given can be somewhat "streamlined" in appearance--for instance, having all summations start with index zero.)

This completes our earlier "partial/half" answer to the question, which lacked the new interesting (less tractable, perhaps) expression for $v(b,\mu)$. My apologies for this somewhat awkward process of answering the posed question. (I considered simply replacing/editing the original answer, but that already had several comments pertaining to it--which might be somewhat confusing for subsequent comments.)

10more comments