The integral in question is given by,

$$I = \int_{0}^1 t^{-a} (1-t)^N (1-xt)^{-a}F[-a,k-a-1/2,k-a;4xt(1-xt)] \, \mathrm{d}t$$

where $F$ denotes the standard hypergeometric series which is only well-defined for $k-a\geq 0$, and the Pochhammer symbol only terminates if either, $a \geq 0$ or $k-a-1/2 \leq 0.$ Expanding the series yields,

$$F = \sum_{n=0}^\infty \frac{(-a)_{n} (k-a-1/2)_n}{(k-a)_n} \frac{(4xt(1-xt))^n}{n!}$$

where $(q)_n$ denotes the rising Pochhammer symbol (contrary to popular notation). We approach the original integral $I$ by integrating term by term as we slowly expand $F$. The case $n=0$ is,

$$I_{0}=\int_{0}^1 t^{-a}(1-t)^N (1-xt)^{-a} \, \mathrm{d}t$$

as the hypergeometric series is unity for $n=0.$ Notice the integral $I_0$ is precisely an Appell series, i.e.

$$I_0= \frac{\Gamma (1-a)\Gamma(N+1)}{\Gamma(N+a+1)}\mathcal{F}[(1-a),a,0,(N+a+1);x,0]$$

where $\mathcal{F}$ denotes the Appell series, rather than the hypergeometric series. We may proceed similarly for the subsequent $n=1$ contribution, namely,
$$I_1 = 4x\frac{a^2(1-a)(k-a-1/2)^2(k-a+1/2)}{(k-a)^2(k-a-1)}\int_0^1 t^{-a+1}(1-t)^N (1-xt)^{-a+1}$$

Luckily $I_1$ is essentially an Appell series also, namely,

$$I_1 = 4x\frac{a^2(1-a)(k-a-1/2)^2(k-a+1/2)}{(k-a)^2(k-a-1)} \frac{\Gamma(2-a)\Gamma(N+1)}{\Gamma(N+3-a)} \\ \times\mathcal{F}[(2-a),(a-1),0,(N+a+1);x,0]$$

Proceed similarly for $n\geq 2$. Goodluck, it's quite a messy problem! Maybe after several iterations you may deduce a general expression for the $n$th contribution $I_n$.