I might as well write an answer with the proof I referenced to above (found as theorem 110 here). Hopefully Darij will write a more detailed answer tomorrow.
The first thing to observe is that your matrix $A$ is the image of the element $$\omega=\sum_{\sigma\in S_n} x^{|\pi|}\pi$$ in the regular representation of the group algebra $\mathbb C[S(n)]$. Next notice that this element factors as $$\omega=(x+J_1)(x+J_2)\cdots(x+J_n)$$ where $J_k$ are the Jucys-Murphy elements defined as $J_k=\sum_{ s < k} (s,k)$.
Let's denote $\Xi_n=\lbrace J_1,J_2,\dots,J_n,0,0,\dots\rbrace$. It is a theorem that for any symmetric function $f\in \Lambda$, the mapping $f\to f(\Xi_n)$ sends symmetric polynomials onto elements of the class algebra $\mathcal Z(n)$.
Now since $f(\Xi_n)\in \mathcal Z(n)$, by Schur's lemma it acts as a scalar on any irreducible representation $V^{\lambda}$ of $\mathbb C[S(n)]$. Jucys theorem says that the central character of $f(\Xi_n)$ acting on $V^{\lambda}$ can be obtained by simply substituting the alphabet $\Xi_n$ with the content alphabet $$A_{\lambda}=\lbrace c(\square): \square\in \lambda\rbrace.$$ SoThese two facts are proved in Jucys' article, "Symmetric polynomials and the center of the symmetric group ring".
So, in particular, the central character of $\omega$ is $$\prod_{\square\in \lambda}(x+c(\square)),$$ and, putting things together, from the decomposition $\mathbb C[S(n)]=\bigoplus_{\lambda \vdash n} (\dim \lambda)V^{\lambda}$, we obtain $$\det(A)=\prod_{k=1}^{n-1}(x^2-k^2)^{r_k},$$ where $$r_k=\sum_{\lambda\vdash n, k\in A_{\lambda}} \dim \lambda.$$