$\newcommand\R{\mathbb R}$This requestThe answer is hopelessno.
E.g., let $A=B=\R$, $B_a=[1-a^2,2]$ for $a\in A$, and $\ell(b)=b^3-3b$ for $b\in B$. Then $\ell(b)$ and $B_a$ are perfectly smooth, but $L$ is not differentiable.
Indeed, here $L(a)=-2+a^4\min(0,3-a^2)$ for $a\in A=\R$. The left derivative of $L$ at $\sqrt3$ is $0$ and the right derivative of $L$ at $\sqrt3$ is $-18\sqrt3$.