I'm not sure there is a definitive answer here, as it's asked if there are "invariants", not a complete characterisation. Further, it's not clear to me if the question is asking about compactness or just whether the WOT and SOT are different.
Indeed, if all we are interested in is to show that the SOT is not the same as the WOT when restricted to the unit ball of $B(H)$, then we don't need to consider compactness at all. As you say, Murphy observes (this is Example 4.1.1 in his book) that the adjoint is WOT continuous but not SOT continuous.
Actually, there is another example which doesn't use the adjoint. We can just consider $S$ the unilateral shift acting on $\ell^2$, and let $S_n = S^n$. Then $S_n\rightarrow 0$ weakly, but of course as each $S_n$ is an isometry, not subnet of $(S_n)$ can converge to $0$ in the SOT.
I don't think it is obvious that the same argument will work for self-adjoint subalgebras $A\subseteq B(H)$. For example, if $A$ is commutative, then the adjoint is continuous for the SOT restricted to the unit ball of $A$.
Let $A=L^\infty(\mathbb T)$ acting on $H=L^2(\mathbb T)$ by multiplication, and let $a_n(e^{i\theta}) = e^{in\theta}$. The Riemann-Lebesgue Lemma shows that $(a_n)$ is WOT convergent to $0$, but clearly $(a_n)$ is not SOT convergent to $0$.
However, let $A=\ell^\infty$ acting on $H=\ell^2$. Then a bounded sequence $(x_n)$ in $A$ is WOT convergent to $0$ exactly when $x_n(k)\rightarrow 0$ for each $k$ (thinking of $x_n$ as a bounded function $\mathbb N\rightarrow\mathbb C$). This implies that $x_n\rightarrow 0$ in the SOT.
Let $X\subseteq B(H)$ be just a subspace. Identify $B(H\oplus H)$ with $2\times 2$-matrices with entries in $B(H)$. Define
$$ Y(X) = \left\{ \begin{pmatrix} \lambda & x \\ 0 & \lambda \end{pmatrix} : x\in X,
\lambda\in\mathbb C \right\}. $$
The $Y(X)$ is a subalgebra, not self-adjoint, and $Y(X)$ is closed in all the various operator topologies if and only if $X$ is closed in that topology. While the unit ball of $Y(X)$ and $X$ do not biject, bounded subsets to correspond to bounded subsets. In this way, we can transport (counter-)examples from subspaces to subalgebras.
Let $X$ be a "row space" of $H$. That is, fixed a unit vector $\xi_0\in H$ and let $X$ be the collection of rank-one operators $\{ \theta_\xi : \xi\in H \}$ where $\theta_\xi:H\rightarrow H; \eta\mapsto (\eta|\xi)\xi_0$. Then $X$ is anti-linearly isomorphic to $H$. A bounded net $(\theta_{\xi_n})$ converges to $x\in B(H)$ in the WOT if and only if
$$ (\eta|\xi_n) (\xi_0|\eta') \rightarrow (x(\eta)|\eta') \qquad
(\eta, \eta'\in H). $$
This is equivalent to $(\xi_n)$ being weakly-convergent in $H$, say to $\xi$, in which case $\theta_{\xi_n}\rightarrow\theta_\xi$. Similarly, $(\theta_{\xi_n})$ converges strongly to $x\in B(H)$ when
$$ \|(\eta|\xi_n) \xi_0 - x(\eta)\|\rightarrow 0 \qquad (\eta\in H). $$
This gives the same condition on $(\xi_n)$. So, the WOT and SOT agree on the bounded subsets of $X$, and hence on the unit ball of $Y(X)$.
