See SGA 4, Exposé VIII, 7.8, which defines an abstract point of a site as a functor from the topos of that site (i.e. the category of sheaves of sets on that site) to the category of sets that commutes with arbitrary inductive (direct) limits and finite projective (inverse) limits.*
One should think of a functor as being the functor that takes a sheaf to its stalk at that point (the "fiber functor" of the point). Since every fiber functor preserves such limits, these are reasonable axioms to postulate.
Grothendieck then proves in this exposé that in the étale site, every functor satisfying the axiom above comes from a (unique) geometric point (up to isomorphism in a suitably-defined sense).
Furthermore, and I can only find this referenced in Brian Conrad's unavailable draft book on the Ramanujan conjecture, in a (classical) topological space where every irreducible subset has a unique generic point (also known as a sober space), such as a normal Hausdorff space or the underlying space of a scheme, every abstract point of the category of sheaves on the topological space corresponds to a unique classical point. This is Proposition 2, Section 3, Chapter IX of Sheaves in Geometry and Logic.
Why characterize points in terms of their functors? This fits in with the general Tannakian formalism, which emphasizes fiber functors, and in particular, defines the fundamental group to be the group of automorphisms of the fiber functor. I.e., we should think of a point as the functor that assigns to a sheaf its stalk at that point.
This also fits into a more general philosophy that we should ignore the site and focus on the topos entirely. One can in fact put a Grothendieck topology on the topos so that it is equal to the category of sheaves on itself.
*Edit: One can also find this in Sheaves in Geometry and Logic, by Maclane and Moerdijk, Chapter VII, Section 5.