$\newcommand\R{\mathbb R}$
Most likely, the integral 
$$I:=\int_0^t dt_1 \int_0^t dt_2 \int_0^t dt_3\, B(t-t_3)F(t_3-t_2)
S(t-t_2)F(t_2-t_1)F(t_1)$$
cannot be expressed in terms of products and convolutions -- because the finite interval $[0,t]$ is not a group (or even a semigroup), and the restrictions $t_2\in[0,t]$ and $t_3\in[0,t]$ are unjustified, since the integrand contains no factors depending only on $t_2$ or only on $t_3$. Since the functions $B,F,S$ are zero on $(-\infty,0)$, the correct(ed) form of the integral $I$ seems to be 
\begin{align*}
	\tilde I&:=\int_\R dt_1 \int_\R dt_2 \int_\R dt_3\, B(t-t_3)F(t_3-t_2)
S(t-t_2)F(t_2-t_1)F(t_1) \\
&=\int_\R {dt_2}\, (B*F)(t-t_2)S(t-t_2)(F*F)(t_2) \\ 
&=\Big(\big((B*F)S\big)*(F*F)\Big)(t),
\end{align*}
so that $\tilde I$ is expressed in terms of products and convolutions. 

---

The similarly corrected integral added later to the OP, 
\begin{align*}
	J&:=\int_\R dt_1 \int_\R dt_2 \int_\R dt_3\,  A(t-t_3)F(t_3-t_2)S(t-t_2)F(t_2-t_1)S(t_3-t_1)F(t_1),
\end{align*}
probably cannot be expressed in terms of products and convolutions. 

A way to check it may be by the following *shift* observation, **which addresses a comment by LSpice**: 

>Suppose we want to show that a function $f$ cannot be expressed in terms of products and convolutions of functions $f_1,\dots,f_n$. To do this, we build all possible binary trees as follows. The nodes of such a tree are nonempty subsets of the set $[n]=\{1,\dots,n\}$. To each node $J$ there corresponds an expression in terms of products and/or convolutions of the functions $(f_j\colon j\in J)$. 
The root of the tree is the set $[n]$. Each node is split into two branches, whose roots are nonempty subsets, which become the nodes next up the tree. Each node is marked either by $p$ (for "product") or $c$ (for "convolution"), depending on whether the daughter expressions of the node get multiplied or convolved. 

>Then the functions $f_1,\dots,f_n$ get shifted horizontally, starting with the leaves of the tree. If two adjacent leaves are the branches of a $p$-node, their corresponding functions get shifted by the same real number, say $u$. After that, the two leaves get deleted and their parent node is marked by the same shift $u$. If two adjacent leaves are the branches of a $c$-node, their corresponding functions get shifted by two arbitrary real numbers, say $u_1$ and $u_2$. After that, the two leaves get deleted and their parent node is marked by shift $u_1+u_2$. This process then continues down to the root. For every $p$-node encountered during this process, the shifts assigned to the daughter branches get equalized, say by imposing the corresponding equality constraint. Finally, at the root of the tree, we check if the function $f$ has the resulting shift property. 

As a comparatively simple case, suppose that a function $f$ can be expressed in terms of products and convolutions of functions $f_1,\dots,f_n$ as follows: 
\begin{equation}
	f=\Big(\prod_{j\in J_1}f_j\Big)*\cdots*\Big(\prod_{j\in J_m}f_j\Big), \tag{1}
\end{equation}
where $\{J_1,\dots,J_m\}$ is a partition of the set $\{1,\dots,n\}$. 
Suppose next that, for each $k=1,\dots,m$ and each $j\in J_k$, the function $f_j$ is shifted horizontally by a real number $u_k$. Then $f$ gets shifted by $u_1+\dots+u_m$. 

For any given functions $f_1,\dots,f_n$, there may be only finitely many representations of $f$ in the form (1); so, we can try and exhaust all of them. 

---

By considering all the distinguishable trees, it is not hard to see that the integral $J$ cannot be expressed in terms of products and convolutions. This can be done even with zero shifts, and even with the additional assumption that $A=S=F$, as shown in this [Mathematica notebook][1]; you can also see the [notebook's pdf image][2]. 

A few comments on that notebook: 

1. E.g., the possible representation $c(F,p(c(F,F),c(F,p(F,F))))$ of the integral $J$ (with $A=S=F$) is the result of the application of the sequence $(c, p, c, c, p)$ of operations $c$[=convolution] or $p$[=product] to the (unlabeled) tree $(x, ((x, x), (x, (x, x))))$, where $x$ represents any leaf. In our case (concerning the integral $J$), there are $6$ trees (each with $6$ leaves) and $2^5$ sequences of operations $c$ or $p$. So, there are $6\times2^5=192$ possible representations of the integral $J$ in terms of products and convolutions. 

2. For positive real $K$ and $s$, let $f_{K,s}:=Kf_s$, where $f_s$ is the density function of the normal distribution with mean $0$ and variance $s^2$. Then for any positive real $K_1,K_2,s_1,s_2$ we have 
$$f_{K_1,s_1}*f_{K_2,s_2}=f_{K,s}\quad\text{for}\quad K=K_1K_2,\ s=\sqrt{s_1^2+s_2^2},$$
$$f_{K_1,s_1}f_{K_2,s_2}=f_{K,s}\quad\text{for}\quad K=\frac{K_1K_2}{\sqrt{2\pi}\sqrt{s_1^2+s_2^2}},\ s=\frac{s_1s_2}{\sqrt{s_1^2+s_2^2}}.$$
So, the family $(f_{K,s})$ is closed with respect to operations $c$ and $p$. 

3. So, taking e.g. $F=f_{1,1}$, we see that each possible representation of the integral $J$ in terms of products and convolutions (such as e.g. $c(F,p(c(F,F),c(F,p(F,F))))$) is of the form $f_{K,s}$ and thus is determined by the corresponding pair $(K,s)$ of positive real numbers. The integral $J$ for $F=f_{1,1}$ is also of the form $f_{K,s}$, actually with $(K,s)=(\frac{1}{8 \pi ^{3/2}},\sqrt{2})$. 

4. The direct calculation done in the mentioned Mathematica notebook shows that none of the $192$ pairs $(K,s)$ corresponding to the $192$ possible representations of the integral $J$ in terms of products and convolutions equals $(\frac{1}{8 \pi ^{3/2}},\sqrt{2})$. So, the integral $J$ cannot be expressed in terms of products and convolutions.

[1]: https://u.pcloud.link/publink/show?code=XZ7V5BXZU62NRsnkzXX85yXVcMQLRuE8arBy

[2]: https://u.pcloud.link/publink/show?code=XZwJ5BXZQ7xxQazlgjjkAfhWJP3gF5THeSHV