In pg. 24 of his book on Galois cohomology, Serre gives the following exercise:
"Give an example of an extension $1 \to P \to E \to G \to 1$ of profinite groups with the following properties:
(i) $P$ is a pro p-group
(ii) $G$ is finite.
(iii) A Sylow p-subgroup of $G$ lifts to $E.$
(iv) $G$ does not lift to $E.$"
I am not sure how this can happen, allow me to explain. Let us start by assuming that all groups involved are finite. Suppose that $S$ is a Sylow $p$-subgroup of $G$ and that it lifts to $E.$ Taking the fiber product along the inclusion $S \to G,$ we get the exact sequence
$$1 \rightarrow P \rightarrow E \times_G S \to S \rightarrow 1.$$ By the assumption that $S$ lifts to $E,$ this exact sequence is split.
If $P$ is an abelian $p$-group, this should imply that there is a lift of $G$ to $E,$ since the map $res:H^2(G,P) \to H^2(S,P)$ is injective by the assumption that $S$ is a Sylow $p$-subgroup.
Now, it seems to me that using the above argument for when $P$ is an abelian $p$-group, we should be able to do some devissage argument to show that given any extension $1 \to P \to E \to G \to 1$ of profinite groups and a lift of a Sylow $p$-subgroup of $G$ to $E$ as in the exercise, there is a lift of $G$ to $E$ as well. But this is in contradiction to the exercise.
Obviously, this means I have made a mistake in my thinking, but I have spent some time on it but can not seem to find it. Could someone help me finding where this mistake is?
Update I now see where my mistake lies. I will post this as an answer tomorrow.