Vertex algebras precisely model the structure of "holomorphic one-dimensional algebra" -- in other words, the algebraic structure that you get if you try to formalize the idea of operators (elements of your algebra) living at points of a Riemann surface, and get multiplied when you collide.
Our geometric understanding of how to formalize this idea has I think improved dramatically over the years with crucial steps being given by the point of view of "factorization algebras" by Beilinson and Drinfeld, which is explained (among other places :-) ) in the last chapter of my book with Edward Frenkel, "Vertex algebras and algebraic curves" (second edition only). This formalism gives a great way to understand the algebraic structure of local operators in general quantum field theories -- as is seen in the recent work of Kevin Costello -- or in topological field theory, where it appears eg in the work of Jacob Lurie (in particular the notion of "topological chiral homology").
In fact I now think the best way to understand a vertex algebra is to first really understand its topological analog, the structure of local operators in 2d topological field theory. If you check out any article about topological field theory it will explain that in a 2d TFT, we assign a vector space to the circle, it obtains a multiplication given by the pair of pants, and this multiplication is commutative and associative (and in fact a commutative Frobenius algebra, but I'll ignore that aspect here). It's very helpful to picture the pair of pants not traditionally but as a big disc with two small discs cut out -- that way you can see the commutativity easily, and also that if you think of those discs as small (after all everything is topologically invariant) you realize you're really describing operators labeled by points (local operators in physics, which we insert around a point) and the multiplication is given by their collision (ie zoom out the picture, the two small discs blend and look like one disc, so you've started with two operators and gotten a third).
Now you say, come on, commutative algebras are SO much simpler than vertex algebras, how is this a useful toy model? well think about where else you've seen the same picture -- more precisely let's change the discs to squares. Then you realize this is precisely the picture given in any topology book as to why $\pi_2$ of a space is commutative (move the squares around each other). So you get a great model for a 2d TFT by thinking about some pointed topological space X.. to every disc I'll assign maps from that disc to X which send the boundary to the basepoint (ie the double based loops of X), and multiplication is composition of loops -- i.e. $\Omega^2 X$ has a multiplication which is homotopy commutative (on homotopy groups you get the abelian group structure of $\pi_2$). In homotopy theory this algebraic structure on two-fold loops is called an $E_2$ structure.
My claim is thinking about $E_2$ algebras is a wonderful toy model for vertex algebras that captures all the key structures. If we think of just the mildest generalization of our TFT story, and assign a GRADED vector space to the circle, and keep track of homotopies (ie think of passing from $\Omega^2 X$ to its chains) we find not just a commutative multiplication of degree zero, but a Lie bracket of degree one, coming from $H^1$ of the space of pairs of discs inside a bigger disc (ie from taking a "residue" as one operator circles another). This is in fact what's called a Gerstenhaber algebra (aka $E_2$ graded vector space). Now all of a sudden you see why people say you can think of vertex algebras as analogs of either commutative or Lie algebra (they have a "Jacobi identity") -- -the same structure is there already in topological field theory, where we require everything in sight to depend only on the topology of our surfaces, not the more subtle conformal geometry.
Anyway this is getting long - to summarize, a vertex algebra is the holomorphic refinement of an $E_2$ algebra, aka a "vector space with the algebraic structure inherent in a double loop space", where we allow holomorphic (rather than locally constant or up-to-homotopy) dependence on coordinates.
AND we get perhaps the most important example of a vertex algebra--- take $X$ in the above story to be $BG$, the classifying space of a group $G$. Then $\Omega^2 X=\Omega G$ is the "affine Grassmannian" for $G$, which we now realize "is" a vertex algebra.. by linearizing this space (taking delta functions supported at the identity) we recover the Kac-Moody vertex algebra (as is explained again in my book with Frenkel).