Not an answer, more like an overgrown comment. The nearest integer to $x$ is the floor of $x+(1/2)$, and you can incorporate the 1/2 into $b$, so you're talking about things like $([ja+b]-[ia+b]+1)/(j-i)$. Now floor is identity minus fractional part, so we get to $(ja+b-\lbrace ja+b\rbrace-ia-b+\lbrace ia+b\rbrace+1)/(j-i)$, which simplifies to $a+(\lbrace ia+b\rbrace-\lbrace ja+b\rbrace+1)/(j-i)$. So you're looking at the minimum of $(\lbrace ia+b\rbrace-\lbrace ja+b\rbrace+1)/(j-i)$ minus the maximum of $(\lbrace ia+b\rbrace-\lbrace ja+b\rbrace-1)/(j-i)$, which may be a little more tractable than what you started out with.
EDIT: As to whether someone has already done this, the paper by Melter, Stojmenovic, and Zunic, A new characterization of digital lines by least square fits, Pattern Recognition Letters 14 (1993) 83-88, available at http://www.site.uottawa.ca/~ivan/F29-digital-line-lsf.PDF may be relevant or may point you to some useful work.
FURTHER EDIT: Another paper that looks like it might be useful is MR2323394 (2008e:68149) Uscka-Wehlou, Hanna, Digital lines with irrational slopes, Theoret. Comput. Sci. 377 (2007), no. 1-3, 157–169.