Jyrki already gave a great answer to many of your questions. So, here's to complement it by addressing a couple points he (she?) didn't cover:
- Hot topic in error-correction is finding LDPC codes with very low "error-floor" for code lengths dozens thoursands bits, this might be useful for optic transmission. However it is not clear for me what kind of math playing role here ?
That depends on the channel (and decoding method) you have in mind. In what follows, we only consider a binary code (because I don't know much about the non-binary case!).
The simplest case is BEC (the binary erasure channel), where stopping sets determine the characteristic of the BER of an LDPC code. Small stopping sets are like small weight codewords in a traditional code; the more your code has small stopping sets, the worse its error correction performance gets in general.
Stopping sets can be described in a completely combinatorial way. A stopping set of a parity-check matrix $H$ is a set of columns in which every row has at least two $1$'s (within the columns). You can rephrase the definition in the language of set systems by regarding $H$ as an incidence matrix; it's equivalent to a full configuration. Or you can reword it by using terms like check nodes and variable nodes as is standard in the LDPC code literature.
Anyway, the smallest stopping sets dominate at sufficiently high SNR. So improving BER directly corresponds to an avoidance problem (or equivalently a forbidden configuration problem) in a binary matrix. So, it's a combinatorial problem that studies a set system avoiding full configurations that gives the desired code length, rate, other additional properties you want such as a particular automorphism. So it's pretty much the same as good ol' algebraic coding theory; avoiding low weight codewords (or equivalently achieving larger minimum distances) is now replaced by avoiding small stopping sets. In fact, the size of a smallest stopping set is called the stopping distance of $H$.
One thing to note is that the notion of stopping distance is defined for a parity-check matrix. Since one same linear code has various different parity-check matrices, you don't say the stopping distance of a code. Other than this, stopping distance is more or less the analogue of minimum distance when it comes to an LDPC code over BEC.
Things are much more complicated for other well-studied channels like everyone's favorite, the AWGN channel, and the coding theory 101 channel, the binary symmetric channel. Some say that in general better stopping distances tend to lead to better BER or at least it's not a bad sign. But this isn't always the case. The kind of sub-structure in $H$ that screws up your decoding algorithm at high SNR for channels other than BEC is typically not easy to describe by simple combinatorics. If you're curious, searching IEEE Xplore with keywords like "trapping sets" and "pseudo-codewords" should direct you to the right papers.
You can also correct quantum errors by taking advantage of the theory of LDPC codes. But I haven't seen a paper that directly studies the analogue of stopping sets, trapping sets, etc. for quantum channels. The main difficulty in the quantum domain is that not every $H$ defines a quantum LDPC code; the rows should correspond to the generators of a stabilizer that is an abelian subgroup of the Pauli group or otherwise you need either a certain number of qubits in the Bell states shared between sender and receiver or assume a very reliable auxiliary quantum channel to "force" your $H$ to define a stabilizer of a larger group. So, you have more things to consider before you can translate the error floor problem of LDPC coding for quantum channels into the language of mathematics. Or you can say the kind of math playing a role for this particular subfield is the ones you need for quantum information on top of the usual math for LDPC codes such as combinatorics, information theory, and probability.
Since you mentioned polar codes in your third question, spatially-coupled LDPC codes are among the hottest classes of LDPC codes that are competing with polar codes. The noise threshold of spatially-coupled ensamble under iterative decoding matches the MAP decoding's threshold over BEC. The quantum analogues of spatially-coupled LDPC codes and polar codes have been/are being studied, too, albeit with a limited success compared to the remarkable progress for the classical case.
- Probably most classical example is the Golay code (1948) and sporadic simple Mathieu groups... [Quote from Wikipedia about how Mathieu groups are automorphism groups of the Golay codes, extended Golay codes, etc. goes here] ...Is this a coincidence, or is there something behind it?
Yes. There's something behind it. For instance, the Mathieu group $M_{24}$ is by definition the automorphism group of the extended binary Golay code, or equivalently, of the Steiner $5$-design of order $24$ and block size $8$, which is called the Witt design in combinatorial design theory. Here is a lecture note on this by A. E. Brouwer. This is one of those interesting interactions between finite simple groups, designs, and codes.
I think this answer is getting too long for a forum post, so for the rest I'll just recommend you search the internet with google or your favorite searching engine by keywords like Golay codes, Steiner designs, Witt designs, simple groups, and the like. That way, I won't upset my previous and current posdoc mentors and Ph.D. supervisor at the same time by writing something stupid about what I should know better.
