This is not true: it does not need to be connected even if $X$ is smooth connected over an algebraically closed field $K$ of characteristic zero. Indeed, if there is an isogeny $\pi \colon A \to B$ and a (necessarily smooth and connected) subscheme $Y \subseteq B$ such that $X = \pi^*Y$ (the scheme-theoretic inverse image), then $a \in A(K)$ fixes $X$ if and only if $\pi(a)$ fixes $Y$, so we get a short exact sequence
$$0 \to \ker \pi \to \operatorname{Stab}_A(X) \to \operatorname{Stab}_B(Y) \to 0.$$
(Exactness on the right follows from surjectivity of $\pi$.) If moreover $\operatorname{Stab}_B(Y)$ is finite, then we see that $\operatorname{Stab}_A(X)$ is a nonzero finite étale algebraic group.
But example of such $X$ are aplenty; for instance we can start with a nontrivial isogeny $\pi \colon A \to B$ and take $Y \subseteq B$ a smooth complete intersection of dimension $1$ of very ample divisors. Then $X$ is a complete intersection of ample divisors [Hart, Exc. III.5.7], so $X$ is connected [Hart, Cor. III.5.7]. Since $Y$ is smooth and $X \to Y$ is finite étale, $X$ is smooth as well (see for instance Tag 01VA). But $Y$ is a curve generating $B$, so it cannot contain a positive-dimensional translate of an abelian subvariety, so $\operatorname{Stab}_B(Y)$ is finite.
Remark. However, it is true that the stabiliser is a smooth closed subgroup of $A$ that is defined over $K$, so its identity component is an abelian variety. Here's how to set this up scheme-theoretically:
We should first say what is being stabilised. I think the natural candidate is the point $X$ in the Hilbert scheme $\mathbf{Hilb}_A$ of closed subschemes of $A$ (see for instance [FGAE, Ch. 5]). The multiplication action $A \times A \to A$ of $A$ on itself naturally defines an action of $A$ on $\mathbf{Hilb}_A$ using the functor of points, and one considers the stabiliser of $X \in \mathbf{Hilb}_A(K)$, i.e. the pullback
$$\begin{array}{ccc}\operatorname{Stab}_A(X) & \to & \mathbf{Hilb}_A \\ \downarrow & & \downarrow \\ A & \to & \mathbf{Hilb}_A \times \mathbf{Hilb}_A,\!\end{array}$$
where the bottom arrow is given on functor of points by $a \mapsto (X,aX)$ and the right vertical map is the diagonal $\Delta_{\mathbf{Hilb}_A}$. (This formula is taken from Milne's notes on algebraic groups [MilneNotes, §9c].)
Then the functor of points point of view immediately shows that $\operatorname{Stab}_A(X)$ is a subgroup functor of $A$, which is a closed subgroup scheme since $\Delta_{\mathbf{Hilb}_A}$ is a closed immersion (see also the remark below). It is smooth since every algebraic group in characteristic $0$ is smooth [MilneNotes, Cor. 10.36] (but a better proof is given in Tag 047N), hence its identity component is an abelian subvariety of $A$ [MilneNotes, Def. 10.13].
Since formation of Hilbert schemes and the pullback square above commute with base change along $\operatorname{Spec} \bar K \to \operatorname{Spec} K$ (see [FGAE, 5.1.5(5)] for the statement about Hilbert schemes), we get
$$\operatorname{Stab}_{A_{\bar K}}(X_{\bar K}) = \operatorname{Stab}_A(X) \underset{\operatorname{Spec} K}\times \operatorname{Spec} \bar K.$$
The Nullstellensatz says that reduced subschemes of $A_{\bar K}$ are uniquely determined by their $\bar K$-points, so $\operatorname{Stab}_{A_{\bar K}}(X_{\bar K})$ could be defined simply as the (set-theoretic) stabiliser of $X_{\bar K}$ in $A_{\bar K}$.
Remark. It's a little dissatisfying that we need properness of $A$ to get representability of the Hilbert scheme, so the same argument doesn't work for affine algebraic groups. But in fact we don't really use representability of $\mathbf{Hilb}_A$, only that the diagonal $\Delta_{\mathbf{Hilb}_A}$ is representable by closed immersions (in the sense of Tags 0023 and 025V). It's possible that this holds for a larger class of algebraic groups, but I don't immediately see whether or not this is true.
References.
[FGAE] B. Fantechi, L. Göttsche, L. Illusie, S. L. Kleiman, N. Nitsure, and A. Vistoli, Fundamental algebraic geometry: Grothendieck’s FGA explained. Mathematical Surveys and Monographs 123. American Mathematical Society (AMS), Providence, RI (2005).
[Hart] R. Hartshorne, Algebraic geometry. Graduate Texts in Mathematics 52. Springer-Verlag, New York-Heidelberg-Berlin (1983).
[Milne] J. S. Milne, Algebraic groups. The theory of group schemes of finite type over a field. Cambridge Studies in Advanced Mathematics 170. Cambridge University Press (2017).
A variant is available online as lecture notes (which I cite because I don't have the book available to me at the moment):
[MilneNotes] J. S. Milne, Algebraic groups. Lecture notes.
