A huge amount of financial mathematics assumes Gaussian distributions of risks and Brownian movement of prices. What efforts have there been to replace these with heavy-tailed distributions? For example, could Black-Scholes be adjusted to assume heavy-tailed distribution of price movements, or is this too mathematically difficult?
A Levy process is a stochastic process with stationary, independent increments. These processes have a lot of structure - partly due to the Levy-Khintchine theorem. This says that the Fourier transform of the transition density of a Levy process has a particularly nice form, and has lead to many applications of fourier theory to finance.
All non-Brownian Levy processes have heavier tails than Brownian motion. They are semimartingales, and thus one can define a sensible notion of stochastic integration with respect to a Levy process. As such, they make nice models for log asset price processes, though they have many other applications in finance.
Take a look at Cont and Tankov's book - it's probably the friendliest introduction to Levy process I've seen. Other more theoretical books include those of Kyprianou, Applebaum and Sato (in approximate order of difficulty).
My answer is that I'm sure it can be done with enough effort, but before you try you should know why you are doing this. Mathematical financial models are used primarily for computing the "fair value" of a financial instrument, which means your best guess of its price, given that there are no directly observable prices of the instrument. The model is calibrated to prices of similar instruments whose prices are observable. So effectively the model is used as an interpolation or extrapolation algorithm. The other major use is in risk management, where scenarios defined by price movements of observable prices are translated using the model into price movements of the more illiquid but related instruments. With all of this, you have to do something you can explain to management, lawyers, accountants, and traders. So it is usually best to work with relatively standard (i.e., Gaussian) models, flaws and all, and make ad hoc adjustments for the model limitations (for example, using volatility skew formulas, etc.). The only firm I've ever heard of using non-Gaussian models systematically for risk is Finanalytica, which was started by a statistician, Doug Martin.
The main barrier is that there is no non-Gaussian model that is widely accepted by the community. Moreover, such a model necessarily has more parameters, and there is no obvious systematic way to calibrate these parameters. If some kind of consensus developed for these two things, then non-Gaussian models would become much more widespread.
On the other hand, if you believe that such a model provides a better fundamental dynamic model of market price movements, then you should be able to use such a model to beat the markets. Here, Jeff Harvey is right in that if anyone has done this, you would have every incentive to keep it a secret. So it is possible that Citadel or Renaissance is using such a model to do proprietary trading, and nobody outside the firm knows about it. I wouldn't know.
Without additional assumptions, the answer is basically no, not in any great generality. The derivation of Black-Scholes requires that you can perfectly hedge movements in the option using a stock and a bond. If the underlying stock price process has jumps, then you have jumps in the value of the option, and you can't hedge those jumps using only two assets. (There is one exception — if the process is Poisson, then you can hedge the jumps, but as soon as you have jumps of more than one size then you're stuck.)
The additional assumption is some rule to determine how the option value jumps when the stock price jumps. One rule is that the jumps are "idiosyncratic risk", and therefore are not hedged. This is called the Merton jump-diffusion model. There's plenty of material online about this model. From a quick Google search, these slides look pretty good.
There is an enormous literature on the subject -- you can start by looking at the standard textbook of Hull
Options, Futures & Other Derivatives with Derivagem CD Value Package (includes Student Solutions Manual for Options, Futuresd Other Derivatives) (7th Edition) by JOHN C HULL (Aug 8, 2008)
where he talks about the jump diffusion processes. You can also look at the very nice book:
A History of the Theory of Investments: My Annotated Bibliography (Wiley Finance) [Hardcover] Mark Rubinstein (Author)
But again, this only scratches the surface of the sea of worms.
As a long time practitioner I suggest some caution in the premise of your question, something which appears to have been overplayed in the popular press. Keep in mind the Levy Doob theorem, for instance, and the fact that arbitrage arguments are not the only structure giving rise to useful mathematics, even in the narrowest definition of this field (which I fear amounts to pricing PDEs in the minds of some, and within that the definition of an infinitesimal generator; and within that, a very small class of diffusion operators - you get my point). This has very limited relevance to credit risk, for example.
Heavy-tailed distributions are already de facto used by almost all financial companies, because they use smiles (different volatilities are assigned to different strikes). For instance in equity options lower and lower strikes are usually assigned higher and higher volatilities. If you back out the implied distribution, you get a distribution with a fat tail.
I think that in Finance there is (at least) two fieds that are directly interested in extending the geometric brownian motion as driver for stock market(or other underlyings)
First is pricing for which I think some good answers have been previously posted
Second is risk management which intends to model the risk factor dynamics over generally short period of time (that lives in real world when pricing is living in risk neutral world). In this field the last say belongs more to statisticians than probabilists, and ways to get stylised facts about market returns (such as fat tails) is typically using extensions of GARCH models
There's been a huge amount of literature written on the subject, Google is full of it. Why heavy tails are not mainstream in basic modeling? That's not because these models are "too complex", but mostly because they are not obvious. And risk management and portfolio techniques need an obvious (linear etc.) model to build on that, otherwise very little can be done for the dollar spent on the analysis. "Heavy tailed" models are a huuuge family, I would compare their difference vs. Gaussian to the difference between processes with memory vs. Markov chains - the two families simply cannot be compared. The first type has thousands of concepts of the second type in it.
Please approach the Stable family with caution - from my experience, it doesn't precisely model real series (but, of course, better than Gauss). A composition (like different laws for the middle term and the tail - like multiplied by some exponential decay, different power laws, etc.) often works better. Of course, you can't toss these distributions around anymore, like you do with Gaussian.