First about the two notions of sheaf: Dugger supposes that his Grothendieck topology is given by a pretopology, i.e. you specify for each object $X$ of the site which families of morphisms $\{U_i \to X\}$ are covering families - from such a covering family you get a sieve $R$ by defining $R(Z)$ as the collection of all those morphisms $Z \to X$ which factor through some morphism from the covering family. Thus you get an objectwise surjective morphism $\coprod_i hom(-,U_i) \to R$. But a morphism $Z \to X$ might factor through several of the $U_i$, but it still only counts as one element of $R(Z)$, hence you want to identify such maps ocurring in several of the coproduct factors: If a morphism $Z \to X$ factors both through $U_i$ and $U_j$, then it also factors through the pullback $U_i \times_X U_j$, i.e. it lies in the image of $hom(-,U_i\times_X U_j) \to hom(-,U_i)$. So we can write $R$ as the coequalizer of $\coprod_{i,j} hom(-,U_i\times_X U_j) \rightrightarrows \coprod_i hom(-,U_i)$.
Thus a morphism $R \to F$ from the sieve to some presheaf $F$ is the same as a cocone of maps from the diagram $\coprod_{i,j} hom(-,U_i\times_X U_j) \rightrightarrows \coprod_i hom(-,U_i)$ to $F$, i.e. a map $\coprod_i hom(-,U_i) \to F$ such that precomposition with both maps from the diagram results in the same morphism $hom(-,U_i\times_X U_j) \to F$. By the universal property of the coproduct the map $\coprod_i hom(-,U_i) \to F$ corresponds to a tuple of maps $hom(-,U_i) \to F$, which in turn by the Yoneda lemma correspond to elements of $F(U_i)$. The commutativity condition for the cone says that your family of elements, i.e. your element in $\Pi_i F(U_i)$, restricts to the same element in $\Pi_{i,j} F(U_i \times_X U_j)$ along both restriction maps. To summarize: A cone under the diagram corresponds uniquely to an element of the equalizer (in sets) of $\Pi_i F(U_i) \rightrightarrows \Pi_{i,j} F(U_i \times_X U_j)$
Now given a morphism $R \to F$, the existence of a lifting to a morphism $hom(-,X) \to F$ corresponds to saying that every such compatible family comes by restriction from $F(X)$, i.e. that the map from $F(X)$ to the above equalizer (given by restriction along the morphisms of the covering family) is surjective. The uniqueness of such a lifting says that it is injective. This summarizes how the two notions of sheaves correspond to each other.
You could say that a sheaf is a presheaf $F$ from who's point of view the diagram $\coprod_{i,j} hom(-,U_i\times_X U_j) \rightrightarrows \coprod_i hom(-,U_i) \to hom(-,X)$ is a coequalizer diagram for any covering family (meaning that mapping out of it into $F$ produces an equalizer diagram).
Now if you want to enforce that some presheaf becomes a sheaf, you have to ensure uniqueness an existence of such liftings. The functor $\mathscr{A}$ ensures uniqueness by identifying elements of $F(X)$ which restrict to the same families in $\Pi_i F(U_i)$ - it is the reflection functor to the subcategory of separated presheaves. The functor $\mathscr{B}$ ensures the existence of a lift. It is defined on presheaves, not on the site, and nothing would be different if your site came from a topological space.
To understand the functor $\mathscr{B}$ you should again adopt a Yoneda reading of the diagram on page 14 of Dugger's article. A compatible family is the same as a map from the "colim" ocuring in that diagram to $F$. But it might not lift to a map from $Hom(-,X)$ to $F$. Thus you replace $F$ by the pushout $\mathscr{B}(F)$; now you have a lifting. The coproducts on the left hand side just mean that you do this with a lot of compatible families at once. But now your new functor $\mathscr{B}(F)$ might no longer be a separated presheaf: Compared to the old $F$ we just added blindly a new element to $F(X)$ for every compatible family in $\Pi_i F(U_i)$ for any covering family. Two different such elements of $F(X)$, coming from two different covers, might actually restrict to the same compatible family in a common refinement of the two covers. This we have to apply $\mathscr{A}$ again. The resulting functor might again lack liftings and so you reiterate until things stabilize. Seen like this, Dugger's construction is rather intuitive I would say.
That you get a lifting property through the formation of such a transfinite colimit is called the "small object argument". This is very clearly and thoroughly exposed in section 2.1 of Mark Hovey's book "Model categories".
Another place where sheafification is done by first creating a separated presheaf, then creating elements corresponding to compatible families are these
notes by Moerdijk (Theorem 1.1). However, in the second step he already adds equivalence classes of compatible families, thus avoiding the transfinite process.
Dugger gives his construction, I would guess, not just to prove the existence of the sheafififcation functor, but rather to introduce the small object argument which works in much more general contexts (see Hovey's book). In particular in certain model categories one gets "fibrant replacement functors" which implement sheafification for presheaves of $\infty$-groupoids (instead of sets).
