In order to present affine spaces in this fashion you need a variation of operads called clones or cartesian operads (which eventually are equivalent to Lawvere theories). You can find some references on nLab.
A clone is like a symmetric operad, but with all maps between finite sets acting not just bijections. However, it is more convenient to phrase the definition as follows. A clone is a functor $C \colon \mathsf{FinSet} \to \mathsf{Set}$ equipped with a family of substitution operators, i.e. for each pair of finite sets $S$ and $T$ we specify a function $(C S)^T \times C T \to C S$ and distinguish projection operations $\pi_s \in C S$ for each $s \in S$. These are supposed to satisfy some axioms (see below).
I will call the clone for affine spaces $\mathbb{A}$. (And I will refrain from using vocabulary from probability theory, this would go better with convex spaces.) For each $S$, $\mathbb{A} S$ is the set of formal affine combinations of elements of $S$, functoriality of $\mathbb{A}$ is given by grouping coefficients of points together when they are identified by a function between finite sets. Substitution operations evaluate affine combinations of affine combinations. If I have a combination $b \in \mathbb{A} T$ and a family $a \colon T \to \mathbb{A}S$, I will denote the resulting substitution by $a \bullet b \in \mathbb{A}S$. (It is really just a shorthand for $s \mapsto \sum_{t \in T} a_{t, s} b_t$.) Moreover, $\pi_s = e_s$ is the trivial combination with coefficient $1$ at $s$ and $0$ elsewhere.
We can now give the following definition. An affine space is a set $E$ equipped with operations $E^S \times \mathbb{A}S \to E$, $(x , a) \mapsto x \bullet a$ subject to the following axioms.
- For every finite set $S$ and all $x \colon S \to E$ and $s \in S$, $x \bullet e_s = x_s$.
- For every pair of finite sets $S$ and $T$ and all $x \colon S \to E$, $a \colon T \to \mathbb{A} S$ and $b \in \mathbb{A} T$, $x \bullet (a \bullet b) = (x \bullet a) \bullet b$.
- For every function between finite sets $\phi \colon S \to S'$ and all $x \colon S' \to E$ and $a \in \mathbb{A} S$, $(x \phi) \bullet a = x \bullet (\phi a)$.
In particular, $\mathbb{A} T$ for a fixed $T$ carries a standard structure of an affine space given by the structure maps of $\mathbb{A}$. In this case, the axioms specialize to some standard facts about formal affine combinations and they are in fact the same as axioms saying that $\mathbb{A}$ is a clone in the first place. This way you can recover the full definition of a clone which I neglected to write down above.