As hinted in Alex Kruckman's comment, it is easier to work with
$$N'(a) = \left|\{x\in \{1,\dots,a-1\} : \sqrt{x(a-x)}\in \mathbb{Z}\}\right|,$$
which is the same as $N(a)$ up to a factor of $2$.
It is straightforward to see that $N'(a)$ equals the number of representations
$$a=d(y^2+z^2),\qquad \gcd(y,z)=1,$$
the connection being $d=\gcd(x,a-x)$, $x=dy^2$, $a-x=dz^2$, $\sqrt{x(a-x)}=dyz$.

The number of ways a number $n$ can be written as a sum of two coprime squares is a multiplicative function $r(n)$ given by $r(2)=1$, $r(p^k)=2$ when $p\equiv 1\pmod{4}$, and $r(p^k)=0$ for all other prime powers $p^k$. So $r(n)$ is quite similar to the divisor function $\tau(n)=\sum_{n=de}1$, and $N'(n)$ is quite similar to the generalized divisor function $\tau_3(n)=\sum_{n=def}1$. In particular, the maximal order of $N'(n)$ is much larger than $\log n$, namely $\exp((c+o(1))\log n/\log\log n)$ for some constant $c>0$ (the peak values occur for $n$'s which are products of distinct primes congruent to $1$ modulo $4$). I am pretty sure the constant equals $c=\log 3$ as in the case of $\tau_3(n)$, but I have not verified this formally.