These three ensembles are hermitian matrices over a (finite dimensional real) field of numbers, and it is known that the only finite dimensional real fields are the real numbers, the complex numbers ($2$-dimensional) and the quaternionic numbers ($4$-dimensional). Octonions are not a field of number since you do not have associativity. The motivation in physics comes from the fact that an hermitian matrix represents a finite dimentional Hamiltonian (an Hermitian operator) in quantum mechanics (then you add randomness, in order to take in account the lack of information about your system, and you let the size of the matrix, that is the dimension of your state space where your Hamiltionian is acting on, going to infinity). In this setting, $N\times N$ quaternionic matrices have to be seen as subclasses of complex hermitian matrices (but of size $2N\times 2N$) and both real symmetric and quaternionic hermitian matrices are a subclass of complex hermitian matrices, with extra symmetries. Anyway, you may imagine many different matrix models relevant for studying (look for Wigner matrices, the answer of Beenakker about other symmetries in physics, the generalized $\beta$-ensemble of Edelman, etc ...)