When $\gcd(a,b)>1$, then we can reach $b$ from $a$ by simply repeatedly adding $\gcd(a,b)$.

The least reachable number (LRN) from $a$ is adding its smallest prime divisor, and the greatest "arrivable" number (GAN) to $b$ is subtracting its smallest prime divisor. At least one of $a$ and its LRN is even, and at least one of $b$ and its GAN is even. Since even numbers have $\gcd$ at least $2$, they are reachable when the difference between $a$ and $b$ is "large".

So the full statement is:

$a$ can arrive at $b$ iff one of the following holds:

 1. $\gcd(a,b)>1$
 2. $a$ is even and $a$ is at most the GAN of $b$
 3. $b$ is even and $b$ is at least the LRN of $a$
 4. $a$ and $b$ are both odd, and the LRN of $a$ is at most the GAN of $b$

Sufficiency is constructive and necessity is obvious by definition of LRN and GAN.