I would like to know if the fundamental group of the connected component of a knot space could be non commutative. I am specially interested in the case of $\mathbb{R}^3$, $\mathbb{S^3}$ or some other $3$fold. All the examples I know result to be commutative (torus knots in $\mathbb{S}^3$, the space of unknots…) but I don't find an argument to prove (or disprove) this in general. I am also curious to know (if the previous question was maybe too hard) if this known at least for the space of long knots.
2 Answers
There is a fairly clean statement about which components of the "long knot space" have abelian fundamental group.
$$K_{3,1} = \{ f : \mathbb R \to \mathbb R^3 : f(t)=(t,0,0) \text{ for } t>1 \} \subset Emb(\mathbb R, \mathbb R^3)$$
This is what I call the "long knot space".
As Mike mentioned $Emb(S^1, S^3) = SO_4 \times_{SO_2} K_{3,1}$.
So you can work out the answer in this case, as well, although it will require a bit more legwork.
Let me state the answer for $K_{3,1}$.
As you've noticed, for the unknot, torus knots, and cables of torus knots, the fundamental groups are abelian. In general, the fundamental group of any component of $K_{3,1}$ has a finiteindex subgroup that's a product of pure braid groups. So step one: how do you avoid nonabelian braid groups? Answer: in the satellite decomposition, at no stage can you have a connect sum of three or more knots.
Okay, but if you limit yourself to such "restricted satellites", how do you stay with abelian groups?
As has been noted, the fundamental group of hyperbolic knot components is abelian (free rank 2) in $K_{3,1}$.
But you get some interesting things if you take, for example, the Whitehead double of an invertible knot, like a torus knot. That component has the fundamental group of the Klein bottle cross a circle. A very nice nonabelian group.
If you take a hyperbolic satellite operation with $n$ input knots, the fundamental group of that component has the form
$$\mathbb Z \times \left( \mathbb Z \ltimes_A \left( A_1 \times \cdots \times A_n \right) \right)$$
Where $A$ is a homomorphism from $\mathbb Z$ to a fairly simple automorphism group of the product $A_1 \times \cdots A_n$. The groups $A_1, \cdots , A_n$ are the fundamental groups of the components of the satellite (input) knots in the satellite operation.
So if the input groups $A_1, \cdots, A_n$ were all abelian to begin with, you would get a nonabelian semidirect product if and only if the homomorphism $A$ acts nontrivially on the product.
This happens if and only if the "pattern link" has a symmetry group that is incompatible with the symmetries of the satellite knots. Rather than write it out as a formula, let me give some examples.
Example 1: Whitehead double of an invertible knot, the homomorphism A : Z > Aut(A_1) factors through a faithful representation Z > Z_2 > Aut(A_1).
Example 2: Whitehead double of a noninvertible knot, the homomorphism A is trivial.
Example 3: The thing I call the "Borromean satellite operation"
In this picture the borromean rings are the "pattern". The two trefoils are the satellite knots.
In the pictured case, the homomorphism factors through a faithful rep $\mathbb Z_4 \to Aut(A_1 \times A_2)$. Both $A_1$ and $A_2$ are $\mathbb Z$. The automorphism in this case would rotate the rank2 lattice $\mathbb Z^2$ by $\pi/2$. If the satellites were distinct invertible knots (say a trefoil and a figure8) then the homomorphism would factor $\mathbb Z_2 \to Aut(A_1 \times A_2)$ and it would act by inversion on the individual factors, i.e. it would not interchange them. [Technically I think I need to modify that image slightly, this is what might be called a "twisted" Borromean satellite operation, and I should change that to the untwisted version]
My paper called "An operad for splicing" covers these kinds of issues in the most detail.
So a recursive "closed form" answer is that you get an abelian group if and only if: (1) when you do a hyperbolic satellite operation it must have "restricted symmetries" where the satellite knots themselves have abelian groups (in the sense of spaces of knots  perhaps I should say "motion group"). And (2) you are never allowed to take a connectsum of three knots. And (3) if you take a connectsum of two prime knots, they must be distinct and those prime summands must have abelian motion groups.
This is the quick answer. If you're comfortable with the language in the splicing operad paper I can be more specific. And of course, the start of the induction operation for the above "closed form" is that torus knots and hyperbolic knots have abelian motion groups. I failed to mention cabling operations  on the level of the fundamental group that is cartesian product with $\mathbb Z$, so it preserves abelianness.

$\begingroup$ Thank you very much, Ryan, for your very accurate answer. I will take a careful look at the references you cited in order to gain an insight into the homotopy type of knot spaces. $\endgroup$– X1921Feb 27, 2019 at 10:51
Yes, the fundamental group may be nonabelian.
For the purpose of this question, we may largely move freely between the spaces of long knots and knots in $S^3$:
There is a fibration $$\text{Emb}_v(S^1, S^3) \to \text{Emb}(S^1, S^3) \to S^3 \times S^2,$$ the last term being the unit tangent bundle of $S^3$ and the first term being the space of embeddings with both $f(1)$ and the direction of $f'(1)$ fixed. The fiber is homotopy equivalent to the space of long knots in $\Bbb R^3$; therefore, if the fundamental group of the space of long knots is abelian, so is the fundamental group of the space of all knots. Conversely, the fundamental group of the space of knots can only be abelian if the fundamental group of the space of long knots is abelian or is an extension of an abelian group by $\Bbb Z$.
It is false for the space of long knots, by Theorem 11 of Ryan Budney's "Little cubes and long knots", from which it follows that if $K$ is prime long knot and $K_n$ denotes the $n$fold connected summand of $K$ with itself, then the space of long knots $\mathcal K_{K_n}$ with fundamental group $\pi$ admits a surjection $\pi \to S_n$, coming from the covering space $$S_n \to \mathcal C_2(n) \times \prod_{i=1}^n \mathcal K_K \to \mathcal C_2(n) \times_{S_n} \prod_{i=1}^n \mathcal K_K \simeq \mathcal K_{K_n},$$ where $\mathcal C_2(n)$ is the configuration space of $n$ distinct ordered points in $\Bbb R^2$.
It follows that the space of knots in $S^3$ may also have nonabelian fundamental group. If we had an extension $0 \to \Bbb Z \to \pi \to A \to 0$ for $A$ abelian, then the surjection $f: \pi \to S_n$ would send the cyclic subgroup $\Bbb Z$ to a normal subgroup of $S_n$. This is impossible for $n \geq 5$, as the only nontrivial normal subgroup of $S_n$ is $A_n$ whenever $n \geq 5$, and the alternating group is hardly cyclic! (There is also no cyclic normal subgroup of $S_4$, but it has one more normal subgroup.) Of course, it is also impossible for $f(\Bbb Z)$ to be trivial, as that would imply that $f$ factors as $f: \pi \to A \to S_n$, and the symmetric group is nonabelian.
So the connected sum of a prime long knot with itself $n \geq 3$ times gives an example in the case of long knots, and the connected sum of a prime long knot with itself $n \geq 4$ times gives an example for knots in $S^3$.

$\begingroup$ I didn't look at Ryan's paper, but it seems to me that if one takes a connect sum of a knot with copies of itself $n$ times, the fundamental group should map to $S_n$ by "permuting" the order of composition of the knots. So do you want your knots $K_i$ to be the same knot type? $\endgroup$– Ian AgolFeb 25, 2019 at 23:24

$\begingroup$ @IanAgol Sorry, this was very sloppy of me. I do want the knots to be of the same type. Ryan takes care to point out that the appropriate subgroup of $S_n$ to quotient by in general is a product of symmetric groups, permuting the knots of the same knot type. The statement I am actually looking at is in a later paper here, Theorem 1.1; I quoted the earlier paper because that is where he proved the relevant result. $\endgroup$– mmeFeb 25, 2019 at 23:30

$\begingroup$ of course as you point out the existence of a surjective homomorphism $\pi \to S_n$ is much easier than what Ryan did to compute the full homotopy type. $\endgroup$– mmeFeb 25, 2019 at 23:34

$\begingroup$ Thank you very much for your answer Mike! It helped quite a lot. I am sorry I cannot choose both answers as the best. $\endgroup$– X1921Feb 27, 2019 at 10:53