I can show that if $X$ is a scheme such that all local rings $\mathcal{O}_{X,x}$ are integral and such that the underlying topological space is connected and Noetherian, then $X$ is itself integral.
This doesn't seem to work without the "Noetherian" condition. But can anyone think about a nice counterexample to illustrate this? So I am looking for a non-integral scheme - with connected underlying topological space - having integral local rings.
Let me try to give a counterexample. (I don't know whether it is 'nice'). First, let us rewrite your properties for an affine scheme $X=Spec(A)$.
Connectedness for $A$ means $A$ has no nontrivial idempotents;
Integrality for $A$ is the usual one ($A$ is a domain);
Local integrality means that whenever $fg=0$ in $A$, every point of $X$ has a neighborhood where either $f$ or $g$ vanishes.
Let us construct a connected locally integral ring that is not integral.
Roughly speaking, the construction is as follows: let $X_0$ be the cross (the union of coordinate axes) on the affine plane. Then let $X_1$ be the (reduced) full preimage of $X_0$ on the blow-up of the plane ($X_1$ has three rational components forming a chain). Then blow up the resulting surface at the two singularities of $X_1$, and let $X_2$ be the reduced preimage of $X_1$ (which has five rational components), etc. Take $X$ to be the inverse limit.
The only problem with this construction is that blow-ups glue in a projective line, so $X_1$ is not affine. Let us correct this by gluing in an affine line instead (so our scheme will be an open subset in what was described above).
Here's an algebraic description:
For every $k\ge 0$, let $A_k$ be the following ring: its elements are collections of polynomials $p_i\in{\mathbb C}[x]$ where $i=0,\dots,2^k$ such that $p_i(1)=p_{i+1}(0)$. Set $X_k=Spec(A_k)$. $X$ is a union of $2^k+1$ affine lines that meet transversally in a chain. (It may be better to index polynomials by $i/2^k$, but the notation gets confusing.)
Define a morphism $A_k\to A_{k+1}$ by $$(p_0,\dots,p_{2^k})\mapsto(p_0,p_0(1),p_1,p_1(1),\dots,p_{2^k})$$ (every other polynomial is constant). This identifies $A_k$ with a subring of $A_{k+1}$. Let $A$ be the direct limit of $A_k$ (basically, their union). Set $X=Spec(A)$. For every $k$, we have a natural embedding $A_k\to A$, that is, a map $X\to X_k$.
Each $A_k$ is connected but not integral; this implies that $A$ is connected but not integral. It remains to show that $A$ is locally integral.
Take $f,g\in A$ with $fg=0$ and $x\in X$. Let us construct a neighborhood of $x$ on which one of $f$ and $g$ vanishes. Choose $k$ such that $f,g\in A_{k-1}$ (note the $k-1$ index). Let $y$ be the image of $x$ on $X_k$. It suffices to prove that $y$ has a neighborhood on which either $f$ or $g$ (viewed as functions on $X_k$) vanishes.
If $y$ is a smooth point of $X_k$ (that is, it lies on only one of the $2^k+1$ lines), this is obvious. We can therefore assume that $y$ is one of the $2^k$ singular points, so two components of $X_k$ pass through $y$. However, on one of these two components (the one with odd index), both $f$ and $g$ are constant, since they are pullbacks of functions on $X_{k-1}$. Since $fg=0$ everywhere, either $f$ or $g$ (say, $f$) vanishes on the other component. This implies that $f$ vanishes on both components, as required.
Hochster has an elegant construction which associates a commutative ring to each infinite totally ordered set with the property that strictly between two distinct elements there is a third one.
The spectrum of such a ring is a connected affine scheme of dimension one, all the local rings of which are domains. The ring itself, however is NOT a domain. So, every ordered set with the property mentioned above yields a scheme with the required property.
Here is the link to Hochster's (one-page) construction
http://www.math.lsa.umich.edu/~hochster/614F08/ECdom.sol.pdf
