I think most people just mentally have in mind the argument you give.
In my thesis I actually wrote this down semi-carefully (including the case l = p, in which case what you want to say is that E[l] is finite over Zp, where E is now the Neron model of your elliptic curve over Q_p.) Or rather I wrote down the direction "l divides Delta => unramified" in Corollary 1.2 of the short version of my thesis. The goal of the thesis, by the way, was to extend this assertion to abelian varieties with real multiplication; the point being that it's not obvious what's supposed to play the role of Delta.