One can find some lecture on modern development of pharma-drugs (lecture  was at public lectorium so it is quite understandable and interesting (imho)):

http://polit.ru/article/2011/03/22/cancercure/

Let me try to sketch a part and mention what is math-related.
(Sorry, lecture is in Russian (try google.translate), but there are some slides inside in English, see also link to "Novartis" below). 

----
Drug development process
----

Math can be used at step 3 (as far as I understood). Let me first give  all steps.


1) Target discovery (find a protein or cascade or smth which are critical for cancer development)

2) Hit discovery (by brute force test 50.000-1.000.000 molecules whether they can kill target or not) 

3) Lead-optimization - (assume on previous step you find "something" which can hit cancer,
but you must care about that this "something" will not kill person also or it is stable enough to work in real life. At this stage one looks for certain modifications which
can preserve the positive features and dismiss negative).

4) Trials on animals 

5) Phase 1 trials (10-50 people just to test that they will not be killed by side effects)

6) Phase 2 trials (100-300 people determine dosation, effectiveness, safety)

7) Phase 3 trial (1000-3000 people determine: side effects, interaction and comparing with other drugs, 

8) Registration

9) Post launch studies

----


This is based on presentation from "Novartis", part can be found at:

http://www.novartis.com/innovation/research-development/drug-discovery-development-process/index.shtml 

"Novartis" is in particular famous for recent innovation of "Gleevec"(=Imatinib) 

http://en.wikipedia.org/wiki/Imatinib 

which is one of rare successes in the field of cancer drugs.)

----

So, (as far as I understood) at step 3 -  "lead optimization" certain mathemaical modelling is sometimes possible. There is certain mathematical-based software which can try to predict some properties of moleculas based on their structure - so when people try
to modify hiting molecula they sometimes use it.