Here is a related construction, but which I find more elementary. The idea is to start from the lamplighter group
$$L_2 := \langle a,t \mid a^2=1, [t^nat^{-n},a]=1 \ (n \in \mathbb{N}) \rangle,$$
to fix an arbitrary subset $I \subset \mathbb{N}$, and to define the new group
$$G_I:= \left\langle a,t,z \mid a^2=z^2=[z,a]=[z,t]=1, [t^nat^{-n},a] = \left\{ \begin{array}{cl} 1 & \text{if } n \in I \\ z & \text{if } n \notin I \end{array} \right. \right\rangle.$$
In other word, we add a central element of order two and we use it to "twist" the commutator relations of $L_2$. By killing $z$, we recover $L_2$ as a quotient, and the corresponding kernel is $\langle z \rangle$. Observe that $z$ has order two in $G_I$ since, by killing $a$ and $t$, it is sent to a non-trivial element in the quotient $\mathbb{Z}/2 \mathbb{Z}$. In other word, we have a central extension
$$1 \to \mathbb{Z}/2\mathbb{Z} \to G_I \to L_2 \to 1.$$
As explained by Yves in his answer, this implies that all the $G_I$ share a common Cayley graph. It remains to show that there exist uncountably many distinct groups among the $G_I$. 

**Lemma:** *The groups $G_I$ and $G_J$ are isomorphic iff $I=J$.*

**Sketch of proof.** First, we need to know that the automorphism group of $L_2$ is generated by the inner automorphisms, the inversion of $t$, and the "transvections" induced by
$$\left\{ \begin{array}{ccc} a \mapsto a \\ t \mapsto gt \end{array} \right., \ g \in \langle \langle a \rangle \rangle.$$
Next, we observe that they all extend to automorphisms of $G_I$. This implies that, if there exists an isomorphism $\varphi : G_J \to G_I$, then, up to post-composing with an automorphism of $G_I$, we can suppose without loss of generality that $\varphi$ induces the identity $L_2 \to L_2$ when we quotient by the centers of $G_I,G_J$. Consequently, there exist $\epsilon,\eta \in \{0,1\}$ such that 
$$\varphi : \left\{ \begin{array}{ccc} z_J & \mapsto & z_I \\ a_J & \mapsto & z_I^\epsilon a_I \\ t_J & \mapsto & z_I^\eta t_I \end{array} \right..$$ 
It follows that $\varphi \left( [t_J^na_Jt_J^{-n},a_J] \right) = [t_I^na_It_I^{-n},a_I]$ for every $n \in \mathbb{N}$, hence $I=J$. $\square$

**Remark:** Following Yves' edition of his answer, we deduce that the lamplighter group $L_2$, which has solvable word problem, is quasi-isometric a group with unsolvable word problem. More explicitly, if $I \subset \mathbb{N}$ is not a computable subset, then $G_I$ is a finitely generated group that is quasi-isometric to $L_2$ but whose word problem is unsolvable.