You can, if you use a slightly more general notion of gluing. (The notion of gluing you present is "wrong", or at least simplistic, in roughly the same way that it is "wrong" to require that a basis for a topology be closed under intersections. E.g., if you do this, then the set of open balls in $\mathbb{R}^n$ for $n > 1$ does not form a "basis.")
Let $X$ be a scheme. Consider the diagram whose objects are open affine subschemes of $X$, and whose morphisms are inclusions $U \hookrightarrow V$ such that $U$ is a distinguished open subset of $V$. Whenever $U$ and $V$ are two objects and $x \in U \cap V$, there exists an object $W \subset U \cap V$ such that $X x \in W$ and $W \hookrightarrow U$, $W \hookrightarrow V$ are both morphisms: Since the distinguished open subsets of $U$ form a basis for the topology, there is a distinguished open $W'$ in $U$ such that $x \in W' \subset U \cap V$. Similarly, there is a section $f$ over $V$ such that $x \in V_f \subset W'$. But then $V_f = W'_f$ is a distinguished open subset of both $U$ and $V$, so we let $W = W'_f$.
It is also not too hard to show that whenever we have a category as above, we can glue things together to form a scheme (i.e., the diagram has a unique colimit in the category of schemes). If someone asks for a precise statement of this, I'll try to cook one up, but it's not particularly nice. (Not quite horrendous, but not very nice either.)
In particular, the fiber product is obtained by gluing together schemes of the form $\mathrm{Spec} A \otimes_C B$, where $\mathrm{Spec} C$ contains the images of both $\mathrm{Spec} A$ and $\mathrm{Spec} B$, with "overlap inclusions" specified by morphisms $A \otimes_C B \to A_f \otimes_C B_g$. An important note here: if $C \to D$ is a ring epimorphism (e.g., corresponds to an open immersion), and $A, B$ are $D$-algebras, then $A \otimes_C B$ is naturally isomorphic to $A \otimes_D B$.

