The following is more a sidenote than an answer to the problem of the fractional iteration of this map, however it introduces a connection to the half-iterate of the exponential/logarithm-map, symmetrizes the iteration and maybe interesting for that.

Define the two functions for the half-exponential and half-logarithm

$$ h(h(x)) = \exp(x)  $$
$$ g(g(x)) = \log(x) $$  

It des not matter which base for the exponentiation we use, say we use base $ b=\sqrt 2$ which allows a real-valued solution for the half-iterate of $ b^x $ and $ log_b(x) $ (I've checked the process using that base and applying "regular fractional iteration")

Then in the original iterated map    
   
$$ a_{k+1} = \frac {a_k+b_k} 2 $$
$$ b_{k+1} = \sqrt {a_k b_k}  $$

substitute the initial $a_0 $ by $ A_0 = h(a_0) $ and $b_0$ by $ B_0 = h(b_0) $     
Then define the iteration      

$$ A_{k+1} = g\left(\frac {h (A_k) + h (B_k)} {2} \right)  $$

$$ B_{k+1} = h(\frac {g (A_k)+g (B_k)} {2})  $$

Then $ g(A_\infty) $ and $ g(B_\infty) $ give the $\operatorname{AGM}(a,b).$

I do not see at the moment how this could be improved to allow a fractional iteration, but perhaps the idea suggests a viable direction.