Suppose $X$ is a smooth scheme, not necessarily projective, over $\mathbb{Z}[1/N]$ for some integer $N\neq 0$. I would like to understand the cohomology groups $H^i_{ét}(X_{\overline{\mathbb{F}}_p}, \mathbb{Q}_{\ell})$ in terms of the singular cohomology of the complex points $X(\mathbb{C})$, but I've run in to some technical issues I don't know how to deal with.

Here is what I know. If we assume additionally that $X$ is projective, then as I understand it there are isomorphisms

$$(*) \ \ \ H^i_{ét}(X_{\overline{\mathbb{F}}_p}, \mathbb{Q}_{\ell}) \cong H^i_{ét}(X_{\overline{\mathbb{Q}}}, \mathbb{Q}_{\ell})$$

induced by specialization provided $p$ does not divide $\ell$. Artin's comparison theorem plus smooth base change then gives isomorphisms $H^i_{ét}(X_{\overline{\mathbb{Q}}}, \mathbb{Q}_{\ell})\cong H^i_{sing.}(X(\mathbb{C}), \mathbb{Q}_{\ell})$ and we're done. The isomorphisms $(*)$ seem very reasonable to me because a smooth proper morphism is something like a fiber bundle in topology, and in that setting any two fibers have isomorphic cohomology.

I have heard that in the non-projective case the isomorphisms $(*)$ still hold provided we omit finitely many primes $p$ (where the set of omitted primes depends on $i$). However, I have not been able to prove this myself or locate a suitable reference. I think I can prove a version for $\mathbb{Z}/\ell^n\mathbb{Z}$ coefficients but I don't see how to get the result upon passage to the limit, since this might involve omitting all primes. I am mostly interested in the case of $X$ a smooth affine scheme over $\mathbb{Z}[1/N]$.

$\textbf{Question}$: If $X$ is a smooth affine scheme over $\mathbb{Z}[1/N]$, are there isomorphisms $H^i_{ét}(X_{\overline{\mathbb{F}}_p}, \mathbb{Q}_{\ell}) \cong H^i_{ét}(X_{\overline{\mathbb{Q}}}, \mathbb{Q}_{\ell})$ for all but finitely many primes $p$?