The Maple 18 code $$ A := eval(convert(hypergeom([-a, k-a-1/2], [k-a], y), FPS, y), y = 4*x*t*(-t*x+1))\,assuming k::posint:$$ $$int(A*t^{-a}*(1-t)^N*(-t*x+1){-a}, t = 0 .. 1)\,assuming a>0,a<1,N::posint,x>0,x<1 $$ outputs the integral under consideration as the series $$\Gamma(N+1)\times$$ $$\sum\limits_{k1=0}^\infty \frac{\mathop{\rm pochhammer}(-a,k1)x^{k1}2^{2k1}{\mbox{$_2$F$_1$}(- k1+a,1-a+ k1;-a+ k1+2+N;x)}}{\mathop{\rm pochhammer}(k-a,k1)\Gamma(-a+k1+2+N)\mathop{\rm pochhammer}(k-a-1/2,k1)^{-1}k1!},$$ where $\mathop{\rm pochhammer}$ is described here. See here for the Maple output.

The Maple 18 code

A := eval(convert(hypergeom([-a, k-a-1/2], [k-a], y), FPS, y), y = 4*x*t*(-t*x+1)) assuming k::posint:
int(A*t^(-a)*(1-t)^N*(-t*x+1)^(-a), t = 0 .. 1) assuming a>0,a<1,N::posint,x>0,x<1;

outputs the integral under consideration as the series $$\Gamma(N+1)\times$$ $$\sum\limits_{i=0}^\infty \frac{\mathop{\rm pochhammer}(-a,i)x^{i}2^{2i}{\mbox{$_2$F$_1$}(- i+a,1-a+ i;-a+ i+2+N;x)}}{\mathop{\rm pochhammer}(k-a,i)\Gamma(-a+i+2+N)\mathop{\rm pochhammer}(k-a-1/2,i)^{-1}i!},$$ where $\mathop{\rm pochhammer}$ is described here. See here for the Maple output.

The Maple 18 code $$ A := eval(convert(hypergeom([-a, 1], [k-a-1/2], y), FPS, y), y = 4*x*t*(-t*x+1))\,assuming k::posint:$$ $$int(A*t^{-a}*(1-t)^N*(-t*x+1){-a}, t = 0 .. 1)\,assuming a>0,a<1,N::posint,x>0,x<1 $$ outputs the integral under consideration as the series $$\Gamma(N+1)\times$$ $$\sum\limits_{k1=0}^\infty \frac{\mathop{\rm pochhammer}(-a,k1)x^{k1}2^{2k1}{\mbox{$_2$F$_1$}(- k1+a,1-a+ k1;-a+ k1+2+N;x)}}{\mathop{\rm pochhammer}(k-a-1/2,k1)\Gamma(-a+k1+2+N)\Gamma(1-a+k1)^{-1}},$$ where $\mathop{\rm pochhammer}$ is described here.

The Maple 18 code $$ A := eval(convert(hypergeom([-a, k-a-1/2], [k-a], y), FPS, y), y = 4*x*t*(-t*x+1))\,assuming k::posint:$$ $$int(A*t^{-a}*(1-t)^N*(-t*x+1){-a}, t = 0 .. 1)\,assuming a>0,a<1,N::posint,x>0,x<1 $$ outputs the integral under consideration as the series $$\Gamma(N+1)\times$$ $$\sum\limits_{k1=0}^\infty \frac{\mathop{\rm pochhammer}(-a,k1)x^{k1}2^{2k1}{\mbox{$_2$F$_1$}(- k1+a,1-a+ k1;-a+ k1+2+N;x)}}{\mathop{\rm pochhammer}(k-a,k1)\Gamma(-a+k1+2+N)\mathop{\rm pochhammer}(k-a-1/2,k1)^{-1}k1!},$$ where $\mathop{\rm pochhammer}$ is described here. See here for the Maple output.