In the following answer I'll focus on the case for general $n$.

Let $m: [0,1] \to \mathbb{C}^{n \times n}$ be measurable and bounded. Let $a \in \mathcal{L}(L^2([0,1]; \mathbb{C}^n))$ be the multiplication with $m$.

**Proposition 1.** Each value in $\partial \sigma(a)$ is an essential spectral value of $a$.

*Proof.* Let $\lambda \in \partial \sigma(a)$ and assume that $\lambda$ is not an essential spectral value. Since $\lambda-a$ can be approximated by invertible operators, it follows from analytic Fredholm theory that $\lambda$ is an isolated spectral value of $a$ and a pole of the resolvent with finite-dimensional spectral space.

Let $p$ denote the corresponding spectral projection; it has finite rank. Since $p$ can be written as a contour integral of the resolvent of $a$, it follows that $p$ is a multiplication operator, too; let $q: [0,1] \to \mathbb{C}^{n \times n}$ denote its symbol. After changing $q$ on a set of measure $0$ if necessary, we may assume that the matrix $q(x)$ is a projection for each $x \in X$.

The set of $x$ for which $q(x) \not= 0$ has non-zero measure (since $p \not= 0$), and thus it follows (similarly as in Christian Remling's answer) that the range of $p$ is infinite-dimensional - a contradiction. $\square$

**Corollary 2.** If $m(x)$ is self adjoint for (almost) every $x \in [0,1]$, then the essential spectrum of $a$ coincides with the spectrum.

*Proof.* Under the assumption of the corollary, the operator $a$ is self-adjoint, so every spectral value of $a$ is in the boundary of the spectrum. $\square$

6more comments