I think one can do something like the following. Let $M = map([0,1], B)$ and $e:M \to B$ be evaluation at 0: this is a Hurewicz fibration and a homotopy equivalence. Now form the pullback fibration $e^*E \to M$, and consider the composite $\bar{E} := e^*E \to M \to B$. This is a fibration fibrewise homotopy equivalent to your original one, and a point in the fibre $\bar{F}_b$ over $b \in B$ is a path $\gamma$ from $b$ to a $b_1$ and point in $p^{-1}(b_1)$. There is an evident $A_\infty$ action of the $A_\infty$ space $\Omega_b B$ on this fibre by composing $\gamma$ with loops at $b$.

Thus there is an $A_\infty$ map $\Omega_b B \to End(F_b)$, which should give you what you want.

Doing the usual Moore loop tricks, one can find an equivalent fibration with an actual action of the grouplike monoid of Moore loops $\Lambda_b B$.