For Question 2, you are asking whether a Sylow $3$-subgroup is normal, and I would expect the fastest and easiest way to do that would be

$\mathtt{G:=SmallGroup(768,k);\ IsNormal(G,SylowSubgroup(G,3))};$

I tried this on $100000$ groups of order $768$ and it took about $2$ minutes CPU time, so the whole calculation should take about 20 minutes. In fact the groups without normal Sylow $3$-subgroups seem all to be at the end of the library of small groups.

For Question 1, if the group is solvable then you could turn it into a $\mathtt{PCGroup}$ if it is not one already and test the normality of a Hall subgroup of the appropriate order.

For non-solvable groups I am not sure. Magma does not seem to have a general command for computing $O^p(G)$. One possibility would be to compute the normal subgroups of the appropriate order $x$ by using

$\mathtt{NormalSubgroups(G:OrderEqual:=x)}$.

Another might be to do it by examining the order of the groups in the lower central series (which can be computed). I would need to do some tests to check which was faster.

For Question 2, you are asking whether a Sylow $3$-subgroup is normal, and I would expect the fastest and easiest way to do that would be

$\mathtt{G:=SmallGroup(768,k);\ IsNormal(G,SylowSubgroup(G,3))};$

I tried this on $100000$ groups of order $768$ and it took about $2$ minutes CPU time, so the whole calculation should take about 20 minutes. In fact the groups without normal Sylow $3$-subgroups seem all to be at the end of the library of small groups.

For Question 1, if the group is solvable then you could turn it into a $\mathtt{PCGroup}$ if it is not one already and test the normality of a Hall subgroup of the appropriate order.

For non-solvable groups I am not sure. Magma does not seem to have a general command for computing $O^p(G)$. One possibility would be to compute the normal subgroups of the appropriate order $x$ by using

$\mathtt{NormalSubgroups(G:OrderEqual:=x)}$.

Another might be to do it by examining the order of the groups in the lower central series (which can be computed). I would need to do some tests to check which was faster.

Added later: After experimenting I think the best general method is to compute the lower central series, which can be done in polynomial time. That also allows you to carry out the test for different primes quickly. Here is some Magma code for this:

IsPNilpotent := function(G,p)
  LC := LowerCentralSeries(G);
  return #LC[#LC] mod p ne 0;
end function;

It is also worthwhile using the method proposed by Geoff Robinson, which is typically very fast when the answer is no, there is no normal $p$-complement.

For Question 2, you are asking whether a Sylow $3$-subgroup is normal, and I would expect the fastest and easiest way to do that would be

$\mathtt{G:=SmallGroup(768,k);\ IsNormal(Sylow(G,3))};$

I tried this on $100000$ groups of order $768$ and it took about $2$ minutes CPU time, so the whole calculation should take about 20 minutes. In fact the groups without normal Sylow $3$-subgroups seem all to be at the end of the library of small groups.

For Question 1, if the group is solvable then you could turn it into a $\mathtt{PCGroup}$ if it is not one already and test the normality of a Hall subgroup of the appropriate order.

For non-solvable groups I am not sure. Magma does not seem to have a general command for computing $O^p(G)$. One possibility would be to compute the normal subgroups of the appropriate order $x$ by using

$\mathtt{NormalSubgroups(G:OrderEqual:=x)}$.

Another might be to do it by examining the order of the groups in the lower central series (which can be computed). I would need to do some tests to check which was faster.

For Question 2, you are asking whether a Sylow $3$-subgroup is normal, and I would expect the fastest and easiest way to do that would be

$\mathtt{G:=SmallGroup(768,k);\ IsNormal(G,SylowSubgroup(G,3))};$

I tried this on $100000$ groups of order $768$ and it took about $2$ minutes CPU time, so the whole calculation should take about 20 minutes. In fact the groups without normal Sylow $3$-subgroups seem all to be at the end of the library of small groups.

For Question 1, if the group is solvable then you could turn it into a $\mathtt{PCGroup}$ if it is not one already and test the normality of a Hall subgroup of the appropriate order.

For non-solvable groups I am not sure. Magma does not seem to have a general command for computing $O^p(G)$. One possibility would be to compute the normal subgroups of the appropriate order $x$ by using

$\mathtt{NormalSubgroups(G:OrderEqual:=x)}$.

Another might be to do it by examining the order of the groups in the lower central series (which can be computed). I would need to do some tests to check which was faster.