$\newcommand\R{\mathbb R}$This request is hopeless. 

E.g., let $A=B=\R$, $B_a=[1-a^2,1+a^2]$ for $a\in A$, and $\ell(b)=b^3-3b$ for $b\in B$. Then $\ell(b)$ and $B_a$ are perfectly smooth, but $L$ is not differentiable. 

Indeed, here $L(a)=-2+a^4\min(0,3-a^2)$ for $a\in A=\R$. The left derivative of $L$ at $\sqrt3$ is $0$ and the right derivative of $L$ at $\sqrt3$ is $-18\sqrt3$.