This is a reasonable thing to do, and is definitely possible in mathjobs--indeed, I believe that we have just offered a job to an individual whose mathjobs-based application had distinct research and teaching letters from the same person. He also had a teaching letter from someone else, as well as a surplus of research letters, consistently with Pete's advice.
I'm not sure how this worked on the letter-writer's end, but on our end the only difference is that there are two files next to the letter-writer's name rather than the usual one. The modifier "(teaching)" which usually appears next to the name of the authors of teaching letters, does not appear next to the name of the letter-writer in question--though maybe this is only because it is quite generally the case that you're only allowed to specify one person as your teaching reference,* and the applicant would have specified the author of the other teaching letter.
*I don't remember whether this is true, but looking at other applications with two teaching letters suggests that it is.
