I will concentrate on the case of odd $q$ and $\xi,-\eta$ being squares, but the solution should be extendable to the remaining cases.
It is convenient to use the language of characters sums in order to compute $N$. To reduce your point counting problem to a character sum problem, the following observation is useful:
$$\#\{ x \in \mathbb{F}_q : ax^2+bx+c=0\} = 1 + \chi(b^2-4ac),$$
where $\chi$ is the unique non-trivial quadratic character of $\mathbb{F}_q^{\times}$, extended to give $0$ on $0$, and $a \neq 0$. This observation is proved by completing the square.
We will need the following well-known formula:
$$(\sim) \, \sum_{x \in \mathbb{F}_q} \chi(x^2+t) = -1$$
for $t\neq 0$. This formula encodes the fact that the number of points on the genus-0 curve $y^2=x^2+t$ is $q-1$. In your case we are definitely lucky, as the computation of $N$ reduces to point counting on genus-0 curves, while for higher genus these point counts are not polynomial in general.
(Cosmetic step) Replacing $(b,d)$ by $(b/\sqrt{\xi},d/\sqrt{\xi})$ we see that we may assume that $\xi = 1$. Further replacing $(c,d)$ by $(c\sqrt{-\eta},d\sqrt{-\eta})$ we see that we may assume that $-\eta = 1$ as well.
Let us expand your defining hypersurface and express it as a quadratic polynomial in $a$:
$$(*) \, a^2(d^2-1) + a(-2bcd) + b^2c^2+b^2-c^2+d^2-1 = 0.$$
The case $d^2 =1$ simplifies to $a(2bcd) = b^2c^2+b^2-c^2$. If $bc \neq 0$, this determines $a$ uniquely. If $bc=0$, we must have $b=c=0$, and $a$ can be arbitrary. This case yields $2((q-1)^2 + q)$ solutions.
Let us assume $d^2 \neq 1$. The discriminant of $(*)$ factorizes as $4(c^2+1-d^2)(b^2+d^2-1)$, which is a lucky coincidence, and possibly the heart the matter. Hence we see that given $b,c,d$ contribute to $(*)$
$$1+\chi( (c^2+1-d^2)(b^2 +d^2-1) )$$
solutions. Summarizing, we have
$$(**)\, N=2(q^2-q+1) + q^2(q-2) + \sum_{d^2 \neq 1, \, b,c,d \in \mathbb{F}_q} \chi( (c^2+1-d^2)(b^2 +d^2-1) ).$$
By $(\sim)$,
$$\sum_{b \in \mathbb{F}_q} \chi( (c^2+1-d^2)(b^2 +d^2-1) ) = \chi(c^2+1-d^2) \sum_{b \in \mathbb{F}_q} \chi(b^2 +d^2-1 ) = -\chi(c^2+1-d^2)$$
when $d^2 \neq 1$, and so
$$\sum_{d^2 \neq 1, \, b,c,d \in \mathbb{F}_q} \chi( (c^2+1-d^2)(b^2 +d^2-1) )= -\sum_{d^2 \neq 1, \, c,d \in \mathbb{F}_q} \chi(c^2+1-d^2),$$
and applying $(\sim)$ once more this becomes
$$-\sum_{d^2 \neq 1, \, c,d \in \mathbb{F}_q} \chi(c^2+1-d^2) = \sum_{d^2 \neq 1} 1 = q-2.$$
Plugging this character sum evaluation in $(**)$, $N=q^3-q$ is obtained, confirming your empirical observation.
For general $\xi$ and $\eta$, a very similar argument will work, because the discriminant of your defining equation (considered as a quadratic polynomial in $a$) still factorizes nicely, specifically it is
$$4(c^2 - \eta-d^2\xi) (-b^2\xi\eta + d^2 \xi + \eta).$$
Even $q$ is easier. The defining equation can now be written as $$(a+1)^2 (\xi d^2 +\eta) = \xi b^2 (c^2+\eta) + c^2.$$ Recall that in $\mathbb{F}_q$ with even $q$, $x \mapsto x^2$ is a field automorphism.
- If $\xi d^2+\eta \neq 0$ (happens for all but a unique $d$), $a+1$ is uniquely determined by $b,c$, giving $(q-1)q^2$ solutions.
- If $\xi d^2+\eta = 0$, $d$ is determined uniquely, $a$ is arbitrary and it remains to count solutions $(b,c)$ to $c^2(\xi b^2+1) = \eta \xi b^2$. Specifying $b$ determines $c$ uniquely, unless $\xi b^2 + 1=0$ (happens for a unique $b$), in which case there are no solutions. So this case contributes $q(q-1)$ solutions.
All in all, $N = (q-1)q^2+q(q-1) = q^3-q$ for even $q$ as well.
