$\newcommand\R{\mathbb R}$This request is hopeless. 

E.g., let $A=B=\R$, $B_a=[1-a^2,1+a^2]$, $\ell(b)=b^3-3b$. Then $\ell(b)$ and $B_a$ are perfectly smooth, but $L$ is not differentiable.