The answer is no and $E=\mathscr{K}(\ell_p)$ for $p\in (1,\infty)\setminus \{2\}$ is already a counterexample. See the proof of Proposition 2.3 in
J. Hennefeld, Compact extremal operators, Illinois J. Math. 21 (1997) 61-65.
Here we identifty $\mathscr{K}(\ell_p)^{**}$ with $\mathscr{B}(\ell_p)$.