$\newcommand{\la}{\lambda}\newcommand{\de}{\delta}\newcommand{\ep}{\delta}$The answer is as follows:
Yes, if the special point is allowed to be not in $C$, and then the moments of inertia of $C$ about all lines through that special point will be the same.
Not in general if the special point is required to be in $C$.
Indeed, the moment of inertia of $C$ about the line $l_{p,a}$ through a point $p=(p_1,p_2)$ orthogonal to a unit vector $a=(a_1,a_2)$ is
\begin{equation*}
M(p,a):=\int_C(a\cdot(x-p))^2\,dx,
\end{equation*}
where $\cdot$ denotes the dot product. So,
\begin{equation*}
M(p,a)=a\cdot G(p)a, \tag{-1}\label{-1}
\end{equation*}
where $G(p)$ is the $2\times2$ matrix with entries
\begin{equation*}
G(p)_{i,j}=\int_C(x_i-p_i)(x_j-p_j)\,dx \\
=\int_C x_i x_j\,dx+p_i p_j \int_C\,dx-p_i\int_C x_j\,dx-p_j\int_C x_i\,dx. \tag{0}\label{0}
\end{equation*}
By shifting and rescaling, without loss of generality (wlog) $\int_C\,dx=1$ and $\int_C x_j\,dx=0$ for $j=1,2$. So,
\begin{equation*}
G=H+p\otimes p,
\end{equation*}
where $H$ is the $2\times2$ matrix with entries $H_{ij}:=\int_C x_i x_j$ and $p\otimes p$ is the $2\times2$ matrix with entries $p_i p_j$.
The matrix $H$ is positive definite. So, by a rotation, wlog $H$ is a diagonal positive-definite matrix, so that
\begin{equation*}
M(p,a)=\la_1 a_1^2+\la_2 a_2^2+(p_1 a_1+p_2 a_2)^2,
\end{equation*}
where $\la_1$ and $\la_2$ are real numbers such that, wlog, $\la_1\ge\la_2>0$.
Fix now a point $p$ and a unit vector $a$, and let
\begin{equation*}
m:=M(p,a).
\end{equation*}
Suppose now that the moment of inertia of $C$ about the line $l_{p,b}$ (through $p$ orthogonal to a unit vector $b=(b_1,b_2)$) is $m$ as as well. This means that $b$ is a solution to the system of equations
\begin{equation*}
\la_1 b_1^2+\la_2 b_2^2+(p_1 b_1+p_2 b_2)^2=m,\quad b_1^2+b_2^2=1. \tag{1}\label{1}
\end{equation*}
Let us say that a unit vector $b$ is good and that the line $l_{p,b}$ is good if $b$ is a solution of system \eqref{1}. Given $p$, the good lines $l_{p,b}$ are in the one-to-one correspondence with the sets of the form $\{b,-b\}$, where $b$ is good. Say that unit vector $b$ and $-b$ are equivalent to each other.
The first equation in \eqref{1} is the equation of an ellipse centered at the origin and the second equation in \eqref{1} is the equation of the unit circle. So, we have either
- Case 1: the ellipse is the same as the circle, and then any line through $p$ is good or
- Case 2: the ellipse is not the same as the circle, and then we have at most two non-equivalent good unit vector $b$'s, and thus at most two good lines.
Clearly, Case 1 occurs only if $p_1 p_2=0$.
So, given the condition $\la_1\ge\la_2>0$, Case 1 occurs only if $p_1=0$ and $p_2^2=\la_1-\la_2$.
However, then both corresponding points $(p_1,p_2)=(0,\pm\sqrt{\la_1-\la_2})$ may be not in $C$. E.g., for small $\de\in(0,1)$, let $C=C^{(\de)}:=[-\frac1{2\de},\frac1{2\de}]\times[-\frac\de2,\frac\de2]$. Then $C$ is convex with the centroid at the origin and with area $1$, the matrix $H$ is diagonal with diagonal entries $\la_1=\frac1{12\ep^2}$ and $\la_2=\frac{\ep^2}{12}$, so that the points $(0,\pm\sqrt{\la_1-\la_2})=(0,\pm\sqrt{1/12}\sqrt{1-\ep^4}/\ep)=:p_\pm^{(\de)}$ are not in this $C=C^{(\de)}$ if $\de$ is small enough. $\quad\Box$
Below one can see the rectangle $C^{(\de)}$ and the points $p_\pm^{(\de)}$ for $\de=1/3$. Then the moments of inertia $M(p_\pm^{(\de)},a)$ about all the lines $l_{p_\pm^{(\de)},a}$ through the points $p_\pm^{(\de)}$ are equal to $\la_1=3/4$, for all (normal) unit vectors $a$.
It follows from \eqref{-1} and \eqref{0} that the moments of inertia $M(p,a)$ depend on $C$ only through the moments $\int_C\,dx$, $\int_C x\,dx$, $\int_C x\otimes x\,dx$ (of the mass distribution over $C$) of orders $0,1,2$. So, the condition of the convexity of $C$ plays hardly any role here.
