4
$\begingroup$

From time to time I find myself wishing to calculate basic statistics on words in the English language. For example, today I found myself wanting a graph of the number of English words vs. their length.

Admittedly, such queries usually arise for me in the context of conversational/recreational purposes, but with the obvious links to cryptography, computational game theory (Scrabble AI etc), and statistics, I think the following question easily falls within the purview of mathematical research.

What good quality resources exist for performing statistical/structural analysis on the set of English words?

One answer to this would be "get a decent word list and write an appropriate program", but, firstly, I don't know what the best word list is and where to get it , and, secondly, in a post-Wolfram Alpha world, I am compelled to search for something higher level, that I can consult from time to time with little set-up needed. For example Mathematica seems to have an elaborate "WordData" package, though I am somewhat unsure of how exhaustive the data set is, given the following excerpt from the Wolfram site:

"Total number of words and phrases in WordData: In[1]:= Length[WordData[All]] Out[1]= 149191"

If anyone has first-hand experience with this package, or even better (as a Maple user), a similar or better resource that is standalone, (or implemented in Maple), then it would be great to hear about it.

Improve this question
$\endgroup$
10
  • 1
    $\begingroup$ I think that this is more of a programming question than a mathematical one and would be more appropriate on SO than MO. $\endgroup$
    – Andrew Stacey
    Jun 16, 2010 at 13:31
  • 1
    $\begingroup$ I considered that too, but decided the kind of answers I want are more likely to be given by those doing mathematical research. I agree it is near the borderline, but I think I'm on the right side. The precedent of this question helped my decison: mathoverflow.net/questions/19046/… $\endgroup$
    – Q.Q.J.
    Jun 16, 2010 at 13:56
  • $\begingroup$ I wasn't all that keen on that question, either! I don't see any actual mathematics or application to mathematics in the question, though. And the latter half is most definitely programming. $\endgroup$
    – Andrew Stacey
    Jun 16, 2010 at 14:24
  • 1
    $\begingroup$ Well I've given my motivations, rationale, and well-defined question, but it seems like we might have different opinions on what a mathematician is allowed to do. So be it :) Please feel free to create and add the mathematical linguistics tag if you deem it appropriate! $\endgroup$
    – Q.Q.J.
    Jun 16, 2010 at 17:27
  • 1
    $\begingroup$ People who think of cryptography, computational game theory, statistics, recreational mathematics, or mathematical thinking in other fields as being mathematics could find the resource I ask about to be useful for doing research in those areas. The question has arisen for me, I think the question is okay here, and this is all I want to say about it. I am okay with the existence of differing opinions. $\endgroup$
    – Q.Q.J.
    Jun 16, 2010 at 18:56

2 Answers 2

Reset to default
1
$\begingroup$

I am very sure this kind ofn things is done in the free (free as in freedom, not as in beer!) statistical language R. .. See www.r-project.org, and then repeat your question on the r-help mailing list, where you are sure to get informed answers about how to do it in R. That it can be done tere, and is done there, is a certainty.

( as for the meta-discussion where this belongs, it belongs on mathoverflow! Statistics is math, not programming.)

Improve this answer
$\endgroup$
1
  • $\begingroup$ The actual question asked is not statistics, though. $\endgroup$
    – Andrew Stacey
    Jun 16, 2010 at 16:58
1
$\begingroup$

The Natural Language Toolkit for Python seems to pretty good.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.