My guess is that this is true, however there is basically no hope in proving this at all in general. The most relevant conjecture here is Manin's conjecture, however what you are asking goes beyond Manin's conjecture. In what follows, I consider the slightly more general and more natural problem of counting the number of rational points on a variety which do not lie on a rational curve.

As an example, take $S$ to be a smooth surface of general type. Then Lang's conjecture predicts that the rational points on $S$ are not Zariski dense. In particular, the closure $Z \subset X$ of the set of rational points consists of a finite union of curves on $S$. The number of rational points on $S$ of height at most $T$ which do not lie on a rational curve is therefore $O( (\log T)^r)$, where $r$ is the maximum of the Mordell-Weil ranks of the elliptic curves contained in $Z$. In particular what you want should be true in this case.

However Lang's conjecture is only known in some very special cases. E.g. it is unknown for smooth surfaces of degree $d \geq 5$ in $\mathbb{P}^3$. For such surfaces one has no idea how to prove sub-polynomial growth away from the rational curves. The best known method for getting upper bounds here is the determinant method of Bombieri-Pila/Heath-Brown/Salberger, however this gives good results for the "worst case", so it does not know the difference between a smooth surface and a singular one. (It can be used to throw away rational curves, but will still give polynomial bounds for the remainder).

The case of K3 surfaces (e.g. smooth quartic surfaces in $\mathbb{P}^3$), is  very interesting as here the rational points are expected to be Zariski dense as soon as there is a single rational point. Here Manin's conjecture predicts that for all $\delta > 0$, there exists an open dense subset $U_\delta \subset S$ such that $U_\delta$ contains $O(T^\delta)$ rational points of height at most $T$. You see that this is close to what you want, however in the conjecture as $\delta$ decreases it says that you need to throw away more and more curves (as $U_\delta$ depends on $\delta$). So you hopefully understand now why I say that what you are asking goes beyond Manin's conjecture.