MathOverflow will be down for maintenance for approximately 3 hours, starting Monday evening (06/24/2013) at approximately 9:00 PM Eastern time (UTC-4).

## Predicting if something is a code

I'm trying to help a non-mathematical friend by posting a question of his here. He studies literature and has come across a book which is written in a made-up language. The book is hundreds of pages, but apparently it is unknown if there is meaning behind what's written. Is there a standard way of predicting whether something like this is written in a code? Obviously we can't know for sure without actually decoding it, but are there some traits to look for? I have no idea, so I thought I would ask here. Thanks!

Here's the book in question: http://en.wikipedia.org/wiki/Codex_Seraphinianus

-
Well, any given sequence $t$ code be a code, since if I could define my code to have the interpretation that $t$ means the Declaration of Independence, and any other string means the United States Constitution. Your question would become more meaningful and interesting if you restricted to certain types of codes and cyphers of a certain size or complexity. What kinds of codes do you allow? – Joel David Hamkins Jul 2 2010 at 15:32
@Joel David Hamkins: I agree. The question looks like that of finding an implementable decision procedure for truth arithmetic. – Sergei Tropanets Jul 2 2010 at 17:01
And should be community wiki. – Sergei Tropanets Jul 2 2010 at 17:01

## 2 Answers

You can start by testing whether or not Zipf's law holds for the text. If it does then it's possible that we're dealing with an artificial "natural language" (not a contradiction in terms I think) that is not cryptographically encoded.

Similarly you can look at the entropy of the text. In particular, you can look at how the probability distribution of a character depends on the previous character. If it appears to have a low entropy (eg. if knowing what one character is allows you to make better predictions about what the next character is) then it's likely that it's not encrypted, or that it's encrypted with a fairly simple cipher. In well encrypted text these kinds of correlations tend to be well hidden.

There is plenty of mathematical discussion relating to the Voynich manuscript. You could probably borrow from that too.

-

Interestingly enough, there was an article about this exact topic (more or less) on slashdot recently. Here's a link to the slashdot story, the scientific article it refers to, and MIT news article about the topic.

The abstract of the scientific article is:

In this paper we propose a method for the automatic decipherment of lost languages. Given a non-parallel corpus in a known related language, our model produces both alphabetic mappings and translations of words into their corresponding cognates. We employ a non-arametric Bayesian framework to simultaneously capture both low-level character mappings and high-level morphemic correspondences. This formulation enables us to encode some of the linguistic intuitions that have guided human decipherers. When applied to the ancient Semitic language Ugaritic, the model correctly maps 29 of 30 letters to their Hebrew counterparts, and deduces the correct Hebrew cognate for 60% of the Ugaritic words which have cognates in Hebrew.

Of course, this assumes that you know what language your unknown language is related to. If it is made up, it's probably worth knowing what languages its author knows and/or speaks.

The tool is probably available from the authors. It does not appear on Benjamin Snyder's web page, though other tools do.

On the other hand, if the text is cryptographically encoded (well) then you'll have a hard job of decrypting it. The field of cryptanalysis or 'code-breaking' deals with that topic.

-