We'll prove that the maximal cardinality of such a set for $M^2\leq N$ has size equal to $$\pi(N)+\sum_{1<n\leq M} \pi(p(n))$$ where $p(n)$ is the smallest prime factor of $n$. Since $$\sum_{1<n\leq M} \pi(p(n))\sim \frac{M^2}{2\log^2 M},$$ this proves that Ilya Bogdanov's example of including all pairwise products of primes not exceeding $M$ is nearly optimal in terms of asymptotics.

As you suggest in the question, this problem is equivalent to constructing the largest subset $A\subset S_M(N)$ such that $\gcd(a,b)\leq M$ for every $a,b\in A$ where $S_M(N)$ is the set of $n\leq N$ whose largest prime factor is at most $M$.

To see why, suppose that $A$ satisfies $\gcd(a,b)\leq M$ for every $a,b\in A$. Then every prime that is greater than $M$ can divide at most one element of $A$. If $p>M$ divides $a\in A$, then making $a=p$ only helps create a larger set $A$. Since these primes do not interact with the $M$-smooth numbers, the proof is complete.

Let $T(N,M)$ denote the maximum size of such a set $A\subset S_M(N)$. Then the size of the largest subset of $[1,N]$ with pairwise $\gcd$'s bounded by $M$ is $$\pi(N)-\pi(M)+T(N,M).$$

As mentioned by Fedja in the comments s, the reasoning above for the primes extends to all integers. By considering those primes $p,q\leq M$ such that $pq>M$, we see that there can only be exactly one such number divisible by $pq$. Similarly for any $p,q,r$ with $pq,qr,rp\leq M$ and $pqr>M$ there can only be one number in our set divisible by $pqr$. Thus we find that the maximal size is the sum over those integers whose largest proper divisor is less than $M$. Since the largest proper divisor of $n$ equals $n/p(n)$ where $p(n)$ is the least prime factor, we can group things based on this, and we have that $$T(N,M)=\sum_{1<p\leq M}\sum_{n\leq M:\ p(n)\geq p}1.$$ Rearranging this equals
$$\sum_{1<n\leq M}\sum_{p\leq p(n)}1=\sum_{n\leq M}\pi\left(p(n)\right),$$ and for composite $n$ $p(n)\leq\sqrt{n}$, so the primes dominate this sum. Thus asymptotically we have $$T(N,M)\sim\sum_{p\leq M}\pi(p)\sim\frac{M^{2}}{2\log^{2}M}.$$

pairwiseproducts, you can throw in products of arbitrarily many (not necessarily distinct) primes $x = p_1 \ldots p_n$, $p_1 \le \ldots \le p_n$, as long as $x/p_1 \le M$. $\endgroup$ – Robert Israel Oct 21 '15 at 18:21