The following is more a sidenote than an answer to the problem of the fractional iteration of this map, however it introduces a connection to the half-iterate of the exponential/logarithm-map, symmetrizes the iteration and maybe interesting for that.
Define the two functions for the half-exponential and half-logarithm
$ h(h(x)) = \exp(x) $
$$ h(h(x)) = \exp(x) $$ $ g(g(x)) = \log(x) $$$ g(g(x)) = \log(x) $$
It des not matter which base for the exponentiation we use, say we use base $ b=\sqrt(2) $$ b=\sqrt 2$ which allows a real-valued solution for the half-iterate of $ b^x $ and $ log_b(x) $ (I've checked the process using that base and applying "regular fractional iteration")
Then in the original iterated map
$ a_{k+1} = \frac {a_k+b_k} 2 $
$$ a_{k+1} = \frac {a_k+b_k} 2 $$
$ b_{k+1} = \sqrt {a_k * b_k} $$$ b_{k+1} = \sqrt {a_k b_k} $$
substitute the initial $a_0 $ by $ A_0 = h(a_0) $ and $b_0$ by $ B_0 = h(b_0) $
Then define the iteration
$ A_{k+1} = g(\frac {h (A_k) + h (B_k)} {2}) $$$ A_{k+1} = g\left(\frac {h (A_k) + h (B_k)} {2} \right) $$
$ B_{k+1} = h(\frac {g (A_k)+g (B_k)} {2}) $$$ B_{k+1} = h(\frac {g (A_k)+g (B_k)} {2}) $$
Then $ g(A_{oo}) $$ g(A_\infty) $ and $ g(B_{oo}) $$ g(B_\infty) $ give the AGM(a,b).$\operatorname{AGM}(a,b).$
I do not see at the moment how this could be improved to allow a fractional iteration, but perhaps the idea suggests a viable direction.
