I am trying to understand the difference between PCA and FA. Through google research, I have come to understand that PCA accounts for all variance, while FA accounts for only common variance and ignores unique variance.
However, I am having a difficult time wrapping my head around how exactly this occurs. I know PCA rotates the axis used to describe the data in order to eliminate all covariance. Does this step still occur in FA? If not, what differentiates FA from PCA? Thanks in advance.
The difference between PCA and FA can be thought of in terms of the underlying statistical models (regardless of estimation methods, although these will change depending on the model used).
Consider $n$ iid observations of a $p$ dimensional (column) vector $X$. Suppose that for each $X_i$, $i \in \lbrace 1, \dots, n\rbrace$, we also had a $k$ dimensional vector $f_i$, with $k \leq p$. These are our "latent factors". A (linear) factor model assumes that $\mbox{E}(X_i \mid f_i) = Bf_i$, where $B$ is a $p \times k$ "factor loadings" matrix and $\mbox{Cov}(X_i \mid f_i) = \Psi$, a diagonal matrix. If we further assume that $\mbox{V}(f_i) = \mbox{I}_k$ so that the factors are independent we see that the marginal covariance is $\Sigma \equiv \mbox{Cov}(X_i) = BB^t + \Psi$.
Roughly, you can think of PCA as making the assumption that $\Psi$ is the zero matrix. In both cases the goal is to find/estimate rotations ($B$) that explain covariance patterns.
If we remove the estimation part of the problem and assume we have $\Sigma$ in hand, the difference is between two ways of decomposing a covariance matrix. We either want a "factor decomposition" $\Sigma = BB^t + \Psi$ or a principle component decomposition $\Sigma = BB^t$.
I think the key really is this: Any covariance matrix will admit either kind of decomposition, but often the rank of $B$ will be substantially smaller if we allow the diagonal elements of $\Psi$ to be non-zero as in the factor decomposition.
Incidentally, finding the factor decomposition for a given covariance that minimizes the rank of $B$ is known as the Frisch problem and is computationally demanding.
PS. I hope this isn't merely a restatement of your remark that "PCA accounts for all variance, while FA accounts for only common variance and ignores unique variance".
