One can find some lecture on modern development of pharma-drugs (lecture was at public lectorium so it is quite understandable and interesting (imho)):
http://polit.ru/article/2011/03/22/cancercure/
Let me try to sketch a part and mention what is math-related. (Sorry, lecture is in Russian (try google.translate), but there are some slides inside in English, see also link to "Novartis" below).
Drug development process
Math can be used at step 3 (as far as I understood). Let me first give all steps.
Target discovery (find a protein or cascade or smth which are critical for cancer development)
Hit discovery (by brute force test 50.000-1.000.000 molecules whether they can kill target or not)
Lead-optimization - (assume on previous step you find "something" which can hit cancer, but you must care about that this "something" will not kill person also or it is stable enough to work in real life. At this stage one looks for certain modifications which can preserve the positive features and dismiss negative).
Trials on animals
Phase 1 trials (10-50 people just to test that they will not be killed by side effects)
Phase 2 trials (100-300 people determine dosation, effectiveness, safety)
Phase 3 trial (1000-3000 people determine: side effects, interaction and comparing with other drugs,
Registration
Post launch studies
This is based on presentation from "Novartis", part can be found at:
"Novartis" is in particular famous for recent innovation of "Gleevec"(=Imatinib)
http://en.wikipedia.org/wiki/Imatinib
which is one of rare successes in the field of cancer drugs.)
So, (as far as I understood) at step 3 - "lead optimization" certain mathemaical modelling is sometimes possible. There is certain mathematical-based software which can try to predict some properties of moleculas based on their structure - so when people try to modify hiting molecula they sometimes use it.