For any sensible definition of perimeter, an adaptation of following argument proves the claim. Consider the nearest-point projection map $p_A:\mathbb R^n\to A$ (that is, for every $x\in\mathbb R^n$, let $p_A(x)$ be the point of $A$ nearest to $x$). It is easy to see that $p_A$ is Lipschitz-1, i.e., $|p_A(x)-p_A(y)|\le |x-y|$ for all $x,y\in\mathbb R^n$. Such a map should decrease perimeters (or your definition of perimeter is flawed), hence $P(X)\ge P(A)$.

With Caccioppoli definition (which I just learned from Wikipedia), you need to translate this argument to the respective language. The resulting proof is e.g. the following. We may assume that $\partial A$ is smooth. Then consider the following smooth unit vector field $\phi$ in $\mathbb R^n\setminus A$:
$$
\phi(x) =\operatorname{grad}(\operatorname{dist}(\cdot,A))(x) .
$$
It is easy to see that $\operatorname{div}\phi\ge 0$ on $\mathbb R^n\setminus A$. Extend $\phi$ to $A$ so that it remains smooth and $|\phi(x)|\le 1$ for all $x\in\mathbb R^n$. Then
$$
P(X) \ge \int_X \operatorname{div}\phi = area(\partial A) + \int_{X\setminus A}\operatorname{div}\phi \ge area(\partial A)
$$
where the first inequality follows from the definition of the perimeter, the equality from Green's formula, and the second inequality from the fact that $\operatorname{div}\phi\ge 0$ on $\mathbb R^n\setminus A$.