$\newcommand{\al}{\alpha}
\newcommand{\de}{\delta}
\newcommand{\De}{\Delta}
\newcommand{\ep}{\varepsilon}
\newcommand{\ga}{\gamma}
\newcommand{\Ga}{\Gamma}
\newcommand{\la}{\lambda}
\newcommand{\Si}{\Sigma}
\newcommand{\thh}{\theta}
\newcommand{\om}{\omega}
\newcommand{\R}{\mathbb{R}}
\newcommand{\Z}{\mathbb{Z}}
\newcommand{\F}{\mathcal{F}}
\newcommand{\E}{\operatorname{\mathsf E}}
\renewcommand{\P}{\operatorname{\mathsf P}}
\newcommand{\ii}[1]{\operatorname{\mathsf I}\{#1\}}
\newcommand{\tf}{\widetilde{f}}$
This infimum is $\frac1\pi$. Indeed, integrating by parts, for any $f\in\F$ we have
\begin{equation}
\frac1{2\pi}=I(f):=\int_0^1 x f(x) dx =-J(f)/2,\quad J(f):=\int_0^1 x^2 f'(x) dx,
\end{equation}
whence
\begin{equation}
H(f):=\int_0^1 x^2 |f'(x)| dx\ge|J(f)|=\frac1\pi,
\end{equation}
so that the infimum in question is $\ge\frac1\pi$.
On the other hand, for any $\ep\in(0,1)$ let
\begin{equation}
f_\ep:=c_\ep g_\ep,\quad g_\ep(x):=\frac x\ep\,1_{(0,\ep]}(x)+\frac{1-x}{1-\ep}\,1_{(\ep,1)}(x),
\end{equation}
where $c_\ep$ is such that $I(f_\ep)=\frac1{2\pi}$, so that $c_\ep\to3/\pi$; the convergence everywhere here is for $\ep\downarrow0$. Then $H(f_\ep)\to\frac1\pi$.
It remains to approximate in $L^1[0,1]$, however closely, both $f_\ep$ and $f'_\ep$ by $f$ and $f'$ for some $f\in\F$; this can be done by using, say, the convolution of $f_\ep$ (or, rather, of a close version of $f_\ep$ with support on $(0,1)$) with a smooth kernel and the multiplication by a constant factor close to $1$. So, the infimum in question is indeed $\frac1\pi$.
