The details of the result slightly stronger than that oulined in Tom Goodwillie's answer appears as Proposition 3.5 of this paper of Bustamante, Krannich, and myself. Let me state a version of this result that uses a bit less notation that the one in the paper:

Let $M$ be a compact smooth manifold of dimension $d$ and $N \subset \mathrm{int}(M)$ be a compact submanifold. If $N$ has handle dimension at most $d-3$ and the fundamental groups of $M$ are finite at all basepoints, then at all basepoints $\pi_k(\mathrm{Emb}(N,M))$ is finitely generated for $k \geq 2$ and polycyclic-by-finite for $k = 1$.

Note that the condition on the handle dimension of $N$ is in particular satisfied if $N$ has codimension $\geq 3$. This proposition is proven by induction over the embedding calculus Taylor tower.

Furthermore, in even dimension $2n \geq 6$ we also have a result in codimension $\leq 2$, but only for the component of the inclusion under a condition on the complement. This is Corollary C of the paper:

Let $M$ be a compact smooth manifold of dimension $2n \geq 6$ and $N \subset \mathrm{int}(M)$ be a compact submanifold. If the fundamental groups of $M$ and $M\setminus N$ are finite at all basepoints, then the groups $\pi_k(\mathrm{Emb}(N,M),inc)$ based at the inclusion are finitely generated for $k \geq 2$.

This is deduced using isotopy extension from a statement about diffeomorphism groups. It uses, among other techniques, embedding calculus and the work of Galatius--Randal-Williams and Friedrich on homological stability for diffeomorphisms of high-dimensional manifolds. You can't drop the condition on the fundamental groups, e.g. the component of the inclusion of $\mathrm{Emb}_\partial(D^{d-2},D^d)$ has infinitely generated homotopy groups for $d \geq 6$.

The details of the result slightly stronger than that oulined in Tom Goodwillie's answer appears as Proposition 3.5 of this paper of Bustamante, Krannich, and myself. Let me state a version of this result that uses a bit less notation that the one in the paper:

Let $M$ be a compact smooth manifold of dimension $d$ and $N \subset \mathrm{int}(M)$ be a compact submanifold. If $N$ has handle dimension at most $d-3$ and the fundamental groups of $M$ are finite at all basepoints, then at all basepoints $\pi_k(\mathrm{Emb}(N,M))$ is finitely generated for $k \geq 2$ and polycyclic-by-finite for $k = 1$.

Note that the condition on the handle dimension of $N$ is in particular satisfied if $N$ has codimension $\geq 3$. This proposition is proven by induction over the embedding calculus Taylor tower.

Furthermore, in even dimension $2n \geq 6$ we also have a result in codimension $\leq 2$, but only for the component of the inclusion under a condition on the complement. This is Theorem C of the paper:

Let $M$ be a compact smooth manifold of dimension $2n \geq 6$ and $N \subset \mathrm{int}(M)$ be a compact submanifold. If the fundamental groups of $M$ and $M\setminus N$ are finite at all basepoints, then the groups $\pi_k(\mathrm{Emb}(N,M),inc)$ based at the inclusion are finitely generated for $k \geq 2$.

This is deduced using isotopy extension from a statement about diffeomorphism groups. It uses, among other techniques, embedding calculus and the work of Galatius--Randal-Williams and Friedrich on homological stability for diffeomorphisms of high-dimensional manifolds. You can't drop the condition on the fundamental groups, e.g. the component of the inclusion of $\mathrm{Emb}_\partial(D^{d-2},D^d)$ has infinitely generated homotopy groups for $d \geq 6$.