Good news and bad news here: This group is connected and Zariski closed for nice general reasons. However, I assume that Dror considered this group because $a(1)$ and $b(1)$ generate a subgroup isomorphic to the dihedral group of order $16$, and are thus relevant to my question. It turns out that the group generated by $a(x)$ and $b(y)$ has dimension at least $9$. The issue is that "the $\mathbb{F}_2$ points of the group generated by $X$" can be much larger than "the group generated by the $\mathbb{F}_2$ points of $X$".
Good news: See Section 2.h of Milne, Algebraic Groups for the following results. Let $G$ be an algebraic group scheme over $k$, let $X$ be a geometrically connected, geometrically reduced, subvariety of $G$ over $k$ and let $\phi: X \to G$ be a map defined over $k$ such that $\phi(X)$ is closed under inversion.
Then there is a smallest algebraic sub-group-scheme $H$ of $G$ such that $\phi$ factors through $H$, and this group is connected. Define $\phi_n : X^n \to G$ by $(x_1, \ldots, x_n) \mapsto \phi(x_1) \cdots \phi(x_n)$, then $\phi^n : X^n \to H$ is dominant for $n$ sufficiently large.
Furthermore (this part isn't in Milne, but seems right to me), let $n$ be large enough that $\phi^n : X^n \to H$ is dominant. Then I claim that $\phi^{2n} : X(k^{\mathrm{alg}})^{2n} \to H(k^{\mathrm{alg}})$ is surjective. Proof: Let $U$ be an nonempty open set within the image of $\phi^n$. Since $H$ is irreducible, for any $h \in H(k^{\mathrm{alg}})$, the open sets $U$ and $U^{-1} h$ have nonempty intersection, so we can find some point $u_1 \in H(k^{\mathrm{alg}})$ which is both in $U(k^{\mathrm{alg}})$ and of the form $u_2^{-1} h$ for $u_2 \in U(k^{\mathrm{alg}})$, and thus $h = u_1 u_2$.
So we can literally think of $H$ as the subgroup of $G$ generated by $X$, as long as we work with points in the algebraic closure.
Bad news: Let $G$ be the group of $7 \times 7$ upper triangular matrices with $1$'s on the diagonal. Define
$$G_k = \{ g \in G : g_{ij}=0 \ \mbox{for} \ i < j < i+2^k \}.$$
So $G = G_0 \supset G_1 \supset G_2 \supset G_3 = \{ \mathrm{Id} \}$.
Each $G_k$ is normal in $G$, and each $G_k/G_{k+1}$ is a product of copies of $\mathbb{G}_a$. Let $H_k = H \cap G_k$. I will verify that $\dim H_0/H_1 = 2$, $\dim H_1/H_2 = 3$ and $\dim H_2/H_3 \geq 2$. So the group $H$ has dimension at least $7$. Details follow. Note that we always have $g_{ij} = g_{(i+2)(j+2)}$ for any $g \in H$.
Computation of $H_0/H_1$ Every product in $H$ is of the form
$$\begin{bmatrix}
1 & x & \ast & \ast & \ast & \ast & \ast \\
0 & 1 & y & \ast & \ast & \ast & \ast \\
0 & 0 & 1 & x & \ast & \ast & \ast \\
0 & 0 & 0 & 1 & y & \ast & \ast \\
0 & 0 & 0 & 0 & 1 & x & \ast \\
0 & 0 & 0 & 0 & 0 & 1 & y \\
0 & 0 & 0 & 0 & 0 & 0 & 1 \\
\end{bmatrix}$$
and every $(x,y)$ occurs, so $H_0/H_1 \cong \mathbb{G}_a^2$.
Computation of $H_1/H_2$ Let $h=\prod_j \left( a(x_j) b(y_j) \right) \in H_1$. Then $\sum x_i = \sum y_j=0$. Then $h_{13} = h_{35} = h_{57} = \sum_{i<j} x_i y_j$ and $h_{24} = h_{46} = \sum_{i \geq j} x_i y_j$. We moreover have $h_{13} + h_{24} = \sum_{i,j} x_i y_j = \left( \sum_i x_i \right) \left( \sum_j y_j \right) = 0$, so $h_{13} = h_{24}$. So we have $h_{13} = h_{35} = h_{57} = h_{24} = h_{46}$. Also, $h_{14} = h_{36}$ and $h_{25} = h_{47}$. So all matrices of $H_1$ are of the form
$$\begin{bmatrix}
1 & 0 & p & q & \ast & \ast & \ast \\
0 & 1 & 0 & p & r & \ast & \ast \\
0 & 0 & 1 & 0 & p & q & \ast \\
0 & 0 & 0 & 1 & 0 & p & r \\
0 & 0 & 0 & 0 & 1 & 0 & p \\
0 & 0 & 0 & 0 & 0 & 1 & 0 \\
0 & 0 & 0 & 0 & 0 & 0 & 1 \\
\end{bmatrix}$$
and $H_1/H_2$ embeds into $\mathbb{G}_a^3$. We compute that $a(x) b(y) a(x) b(y)$ has $(p,q,r) = (xy, x^2 y, x y^2)$ and every vector in $(\mathbb{F}_2^{\mathrm{alg}})^3$ is a sum of vectors of the form $(xy, x^2 y, x y^2)$, so $H_1/H_2 \cong \mathbb{G}_a^3$.
Computation of $H_2/H_3$ At this point, I already know that $\dim H \geq 5$, so it is not a $4$-dimensional unipotent group. I wasn't able to fully compute $H_2$. For $h \in H_2$, we definitely have $h_{15}=h_{37}$, so $\dim H_2 \leq 5$. Every element of $H_2$ that I have found obeys the additional equation $h_{15} = h_{26}$. I can find enough elements to generate the $4$-dimensional subspace cut out by these equations. So $H_2$ is a subgroup of $\mathbb{G}_a^5$ which contains at least $\mathbb{G}_a^4$.
