This might qualify as experimental mathematics fiction.
The Nine Billion Names of GodThe Nine Billion Names of God by Arthur C. Clarke.
monks created an alphabet in which they calculated they could encode all the possible names of god, numbering about 9,000,000,000 ("nine billion") and each having no more than nine characters. Writing the names out by hand, as they had been doing, even after eliminating various nonsense combinations, would take another 15,000 years; the monks wish to use modern technology in order to finish this task more quickly.