Here is a very rough answer.

The Gromov-Witten invariants show up in a few a priori different
contexts within string theory. Let me focus on one particular place they show up that is
directly related to conventional physics, as opposed to topological
quantum field theory.

Type IIA string theory is formulated on a spacetime "background"
which is, in the simplest setup, just a Lorentzian 10-manifold. The
equations of motion of the theory require (at least in their leading
approximation) that the metric on this 10-manifold should be Ricci-flat.

A popular thing to do is to take this 10-manifold of the form
X x R^{3,1}, where X is a compact Calabi-Yau threefold.

We can simplify matters by taking X to be very small ---
smaller than the Compton wavelength of any of the particles we are able
to create. (Remember that in quantum mechanics particles have a
wavelike character, with wavelength inversely related to their energy;
since we only have limited energy available to us, we can't make
particles with arbitrarily short wavelength.) A little more precisely,
let's take X such that the first nonzero eigenvalue of the Laplacian is
larger than the energy scale we can access.

In this case we low-energy
observers will not be able to detect X directly in any experiments. To
us, spacetime will appear to be R^{3,1}.
What will be the physics we see on this R^{3,1}? We will see
various different species of particle. Each species of particle that
we see corresponds to some zero-mode of the Laplacian of X.
In particular, there are particles corresponding to classes in H^{1,1}(X).

The genus 0 Gromov-Witten invariants are giving
information about the interactions between these particles. (So if you want to calculate what will come out when you
shoot two of these particles at each other, one of the inputs to that calculation
would be the genus 0 Gromov-Witten invariants.) The higher genus Gromov-Witten
invariants are giving information about interactions which involve these particles
together with other particles related to the gravitational interaction.