I came up with a proof of my claim when $A$ is bounded, I'll reproduce it here although it is likely that I am the only one interested (since I asked the question). The idea is to use facts from mapping degree theory (this might be elementary for people familiar with this theory, but I discovered it just a few days ago while thinking about this problem).

Let $T>0$ and $A\subset \mathbb{R}^d$ open and bounded. Consider the function $\tilde X: (t,x) \mapsto (t,X_t(x))$, the set
$$
S = \tilde X([0,T]\times \partial A)
$$
which is compact, and its (relative) complement $S^c = ([0,T]\times \mathbb{R}^d)\setminus S$ which is open in the relative topology. Since connected components of $S^c$ are path-connected, it follows, using the invariance under homotopy of the degree (and the other properties of Theorem 1 here) that
$$
(t,x)\mapsto \deg(X_t,A,x)
$$
is constant in each connected component of $S^c$, and more precisely, equals $1$ if the connected component intersects $\{0\}\times A$ and $0$ if it intersects $\{0\} \times (\mathbb{R}^d\setminus A)$. In particular, *no connected component intersects both*. Here, $\deg(f,A,z)$ is the degree of a map $f$ at a point $z$ with respect to a set $A$ (in a nutshell, it gives the "algebraic" number of solutions to $f(u)=z$ for $u\in A$ and in particular, one such solution is guaranteed to exist if $\deg(f,A,z)\neq 0$). Apparently, the use of degree theory is only permitted in bounded sets, hence our assumption on $A$ (a more detailed proof would replace $\mathbb{R}^d$ by a bounded set as an intermediate step).

So far, we have not used the semi-group property: it will be used to show that if $B$ is a connected component of $S^c$ where the above degree vanishes, then $B\cap \tilde X([0,T]\times A)=\emptyset$. To see this, let $x\in A$: if $(X_t(x))_{t\in [0,T]}$ leaves at some point the connected component of $S^c$ containing $x$, then there exists ${t_0}\in [0,T]$ such that $\tilde X_{t_0}(x)\in S$. Then the semigroup property implies $\tilde X_t(x)\in S$ for all $t\geq t_0$ and thus, for all $t\in [0,T]$, $\tilde X_t(x)\notin B$.

The final result follows by taking a section of $S^c$ at a time $t_0\in {]0,T[}$, i.e. considering $\{x \in \mathbb{R}^d \;;\; (t_0,x) \in S^c\}$: it has the property that its boundaries are contained in $X_{t_0}(\partial A)$ and that its connected component are either included in $X_{t_0}(A)$ or in its complement. Moreover, it contains $X_{t_0}(A)$. Thus $\partial X_{t_0}(A) \subset X_{t_0}(\partial A)$.

In the end, our proof tells more about the situation than just this final property. Using the vocabulary from fluid dynamics a summary of the proof would be: mapping degree theory is used to prove that *stream tube walls* do not have *holes*, and the flow property is used to assert that all *stream lines* are confined by these walls.