Some expert (physicist, working partly in finance) recommended me the book:

Jean-Philippe Bouchaud, Marc Potters (2003). Theory of Financial Risk and Derivative Pricing: From Statistical Physics to Risk Management Amazon

It is econophysics approach to analysis of financial markets. It uses quite advanced mathematics including random matrices, stable distributions and so on. One can also look for the papers by these authors in arxiv.

Another expert recommended me the following site: http://www.opentradingsystem.com/quantNotes/main.html about quantative finances,

and the book " High-Frequency Trading. A Practical Guide to Algorithmic Strategies and Trading Systems" IRENE ALDRIDGE