Some expert (physicist, working partly in finance) recommended me the book: 

Jean-Philippe Bouchaud, Marc Potters (2003). Theory of Financial Risk and Derivative Pricing: From Statistical Physics to Risk Management  [Amazon][1]

It is [econophysics][2] approach to analysis of financial markets.  It uses quite advanced mathematics including random matrices, stable distributions and so on. 
One can also look for the papers by these authors in [arxiv][3].

Another expert recommended me the following site:
http://www.opentradingsystem.com/quantNotes/main.html
about quantative finances,

and the book " [High-Frequency Trading. A Practical Guide to Algorithmic
Strategies and Trading Systems][4]"
IRENE ALDRIDGE 


  [1]: http://www.amazon.com/Theory-Financial-Risk-Derivative-Pricing/dp/0521741866/ref=sr_1_fkmr0_1?s=books&ie=UTF8&qid=1359053375&sr=1-1-fkmr0&keywords=Jean-Philippe+Bouchaud%2C+Marc+Potters+%282003%29.+Theory+of+Financial+Risk+and+Derivative+Pricing.+Cambridge+University+Press++%5BAmazon%5D%5B1%5D
  [2]: http://en.wikipedia.org/wiki/Econophysics
  [3]: http://arxiv.org/find/all/1/all%3A+AND+Bouchaud+Potters/0/1/0/all/0/1
  [4]: http://www.amazon.com/High-Frequency-Trading-Practical-Algorithmic-Strategies/dp/1118343506/ref=sr_1_1?s=books&ie=UTF8&qid=1359054076&sr=1-1&keywords=High-Frequency+Trading.+A+Practical+Guide+to+Algorithmic+Strategies+and+Trading+Systems