Your guess is correct. Since $2^{\omega(n)} = \sum_{d | n} \mu^2(d)$ one has
$$
S = \sum_{(a,b,c) \in R(X)} 2^{\omega(4ac-b^2)} = \sum_{d \leq X} \mu^2(d) |R_d(X)|
$$
where $R_d(X)$ is the set of integer triples $(a,b,c) \in R(X)$ such that $4ac \equiv b^2 \pmod d$.

Let $d$ be a squarefree integer. Let $a,b$ be such that $|b| \leq a$ and let $s = \mathrm{gcd}(4a,d)$. Then for $(a,b,c)$ to be in $R_d(X)$ one must have $s | b^2$, or equivalently $s |b$ since $s$ is squarefree, and also $c \equiv (4a/s)^{-1} b^2 \ \mathrm{mod} \ d/s$. The number of integers $c$ like that, with $a \leq c \leq b^2/(4a) + X/(4a^2)$ is

$$
\leq \frac{s}{d}( \frac{b^2}{4a} + \frac{X}{4a^2} - a) + 1 \leq \frac{s}{d}( \frac{X}{4a^2} - \frac{3 a}{4}) + 1\leq\frac{sX}{4 a^2d} + \mathrm{1}_{as \leq 4d/3}.
$$

Moreover the number of integers $b$ such that $|b| \leq a$ and $s | b$ is
$$
\leq \frac{2a}{s} + 2.
$$
Since the constraint $X \geq a(4ac-b^2)$ yields $ad \leq X$, we have
$$
|R_d(X)| \ll \sum_{ad \leq X} (\frac{sX}{a^2d} + \mathrm{1}_{as \leq 4d/3})(\frac{a}{s} + 1) \\
\ll \sum_{ad \leq X} ( \frac{X}{ad} + \frac{sX}{a^2d} + \frac{a}{s} \mathrm{1}_{as \leq 4d/3} + 1 )
\ll \frac{X \log X}{d} + \frac{X}{d} + \sum_{ ad \leq X} ( \frac{sX}{a^2d} + \mathrm{1}_{as \leq d} \frac{a}{s} ).
$$
Summing over $d$ we get
$$
S \ll X \log^2(X) + \sum_{s} \sum_{ad \leq X \\ s|4a , s|d} ( \frac{sX}{a^2d} + \mathrm{1}_{as \leq 4d/3} \frac{a}{s} ).
$$
For each $s$, the inner sum is
$$
\ll \sum_{ad \leq X/s^2 } ( \frac{X}{s^2 a^2d} + \mathrm{1}_{as \leq 16d/3} a )
$$
The first sum is $\ll X s^{-2}$, while in the second we have the constraint $a^2 \ll X s^{-3}$ which yields
$$
\ll \sum_{a \ll X^{1/2} s^{-3/2}} a \ll X s^{-3} + X^{1/2} s^{-3/2}.
$$
So the global sum over $s$ is $\ll X$ and we are done.

Remark: one should be able to extract from this proof an asymptotic formula of the form $c X \log^2(X) + O(X \log X)$.