Though it is not a finite field the non-archimedian versions of quantum mechanics have indeed been used quite a bit in the recent years. Here I would like to draw attention to a different approach, not to the $p$-adic numbers which have been mentioned already in several answers/comments.

One big flaw of the $p$-adics is that they are not ordered: there is no notion of positivity in the field of $p$-adics. Positivity on the other hand is crucial at many places in quantum physics (stability, probabilistic interpretations, ...). So one may wonder whether there are field extensions of $\mathbb{Q}$ which are ordered and still relevant in QM. Of course, if we stick to archimedian orders then the only field (which is also complete) are the reals, so this we all know. However, if we allow for non-archimedian orders then there are plenty more such fields. The most simple ones are perhaps the formal Laurent series $\mathbb{R}((\hbar))$ in a formal parameter $\hbar$. This is the quotient field of the (ordered) ring of formal power series $\mathbb{R}[[\hbar]]$.

As my notion already suggest this is the field underlying quantization theory itself if one (as a first step into the right direction) treats quantization as a (at this stage still) infinitesimal deformation of classical physics with deformation parameter $\hbar$, commonly known as deformation quantization. The non-achimedian order simply means that $\hbar$ is positive but smaller than e.g. and $\frac{1}{n}$ for $n \in \mathbb{N}$. So physically speaking the quantum effects are still "very small" at this stage of the theory.

It is now the positivity which allows to mimick many constructions from operator algebra also in this framework like e.g. a GNS construction, notions of pre Hilbert spaces, etc. Of course, for a honest quantum theory, $\hbar$ is not infinitesimally small, but has to be replaced by a real positive number. From this point of view the formal deformation quantization approach is only a first step, but still a very important one when it comes to the construction of quantum mechanical models for complicated classical theories.

OK, I hope this is not too off-topic, but I was inspired by the nice answers on the $p$-adics ;)

Though it is not a finite field the non-archimedian versions of quantum mechanics have indeed been used quite a bit in the recent years. Here I would like to draw attention to a different approach, not to the $p$-adic numbers which have been mentioned already in several answers/comments.

One big flaw of the $p$-adics is that they are not ordered: there is no notion of positivity in the field of $p$-adics. Positivity on the other hand is crucial at many places in quantum physics (stability, probabilistic interpretations, ...). So one may wonder whether there are field extensions of $\mathbb{Q}$ which are ordered and still relevant in QM. Of course, if we stick to archimedian orders then the only field (which is also complete) are the reals, so this we all know. However, if we allow for non-archimedian orders then there are plenty more such fields. The most simple ones are perhaps the formal Laurent series $\mathbb{R}((\hbar))$ in a formal parameter $\hbar$. This is the quotient field of the (ordered) ring of formal power series $\mathbb{R}[[\hbar]]$.

As my notion already suggest this is the field underlying quantization theory itself if one (as a first step into the right direction) treats quantization as a (at this stage still) infinitesimal deformation of classical physics with deformation parameter $\hbar$, commonly known as deformation quantization. The non-achimedian order simply means that $\hbar$ is positive but smaller than e.g. $\frac{1}{n}$ for all $n \in \mathbb{N}$. So physically speaking the quantum effects are still "very small" at this stage of the theory.

It is now the positivity which allows to mimick many constructions from operator algebra also in this framework like e.g. a GNS construction, notions of pre Hilbert spaces, etc. Of course, for a honest quantum theory, $\hbar$ is not infinitesimally small, but has to be replaced by a real positive number. From this point of view the formal deformation quantization approach is only a first step, but still a very important one when it comes to the construction of quantum mechanical models for complicated classical theories.

OK, I hope this is not too off-topic, but I was inspired by the nice answers on the $p$-adics ;)