For an overview of applications of p-adic numbers in physics I would refer to the two links I added in the comment field, and to this nLab entry. Regarding the second question "What is the most convincing justification in physics that we need to work over the field of real or complex numbers" I would like to quote Freeman Dyson in Birds and Frogs:

When I look at the history of mathematics, I see a succession of illogical jumps, improbable coincidences, jokes of nature. One of the most profound jokes of nature is the square root of minus one that the physicist Erwin Schrödinger put into his wave equation when he invented wave mechanics in 1926. Starting from wave optics as a model, he wrote down a differential equation for a mechanical particle, but the equation made no sense. The equation looked like the equation of conduction of heat in a continuous medium. Heat conduction has no visible relevance to particle mechanics. Schrödinger’s idea seemed to be going nowhere. But then came the surprise. Schrödinger put the square root of minus one into the equation, and suddenly it made sense. Suddenly it became a wave equation instead of a heat conduction equation. And Schrödinger found to his delight that the equation has solutions corresponding to the quantized orbits in the Bohr model of the atom.

It turns out that the Schrödinger equation describes correctly everything we know about the behavior of atoms. It is the basis of all of chemistry and most of physics. And that square root of minus one means that nature works with complex numbers and not with real numbers.

All through the nineteenth century, mathematicians from Abel to Riemann and Weierstrass had been creating a magnificent theory of functions of complex variables. They had discovered that the theory of functions became far deeper and more powerful when it was extended from real to complex numbers. But they always thought of complex numbers as an artificial construction, invented by human mathematicians as a useful and elegant abstraction from real life. It never entered their heads that this artificial number system that they had invented was in fact the ground on which atoms move. They never imagined that nature had got there first.

For an overview of applications of p-adic numbers in physics I would refer to the Wikipedia and Physics.stackexchange links, and to this nLab entry. Regarding the second question "What is the most convincing justification in physics that we need to work over the field of real or complex numbers" I would like to quote Freeman Dyson in Birds and Frogs:

When I look at the history of mathematics, I see a succession of illogical jumps, improbable coincidences, jokes of nature. One of the most profound jokes of nature is the square root of minus one that the physicist Erwin Schrödinger put into his wave equation when he invented wave mechanics in 1926. Starting from wave optics as a model, he wrote down a differential equation for a mechanical particle, but the equation made no sense. The equation looked like the equation of conduction of heat in a continuous medium. Heat conduction has no visible relevance to particle mechanics. Schrödinger’s idea seemed to be going nowhere. But then came the surprise. Schrödinger put the square root of minus one into the equation, and suddenly it made sense. Suddenly it became a wave equation instead of a heat conduction equation. And Schrödinger found to his delight that the equation has solutions corresponding to the quantized orbits in the Bohr model of the atom.

It turns out that the Schrödinger equation describes correctly everything we know about the behavior of atoms. It is the basis of all of chemistry and most of physics. And that square root of minus one means that nature works with complex numbers and not with real numbers.

All through the nineteenth century, mathematicians from Abel to Riemann and Weierstrass had been creating a magnificent theory of functions of complex variables. They had discovered that the theory of functions became far deeper and more powerful when it was extended from real to complex numbers. But they always thought of complex numbers as an artificial construction, invented by human mathematicians as a useful and elegant abstraction from real life. It never entered their heads that this artificial number system that they had invented was in fact the ground on which atoms move. They never imagined that nature had got there first.