Finding correlation in large data, non-numeric sets - MathOverflow

Finding correlation in large data, non-numeric sets

2009-11-18T19:56:41Z

Suppose I collect a lot of data from a group of persons, like

their height
their weight
color of eyes (chosen from eg the four categories blue/brown/black/other)
sex
day of the week the measurement was done

and I want to find which of the above data correlates with height.

For weight, I can image how that goes: just make the (x,y) set and measure stuff like Pearson's correlation. But I get stuck for things like 'colors of eyes' where there's no natural way to order them, or binary data like sex. Also for day of the week (where I'd like to find no correlation at all) but I wouldn't know how to order them.

How do I do this?

Answer by Kim Greene for Finding correlation in large data, non-numeric sets

2009-11-18T20:32:17Z

I think sex can be written as a yes/no question. Female = 1. Male = 0.

You can extend this to eye color. Blue eyes: yes/no. Green eyes: yes/no. Brown eyes: yes/no. One variable like eye color becomes three variables that are either 0 or 1. This gives you a lot of variables but you can do numerical things with them.

Sometimes it makes sense to look at these variables as the probability of being male, having green eyes or whatever.

Answer by Jon Awbrey for Finding correlation in large data, non-numeric sets

2009-11-18T21:18:28Z

There are lots of textbooks under headings like "Categorical Data Analysis", "Non-Parametric Statistics", et cetera, and all the standard statistical packages — GenStat, SAS, SPSS, and other names that will make me sound real ancient if I mention them — will have lots of practical advice about their usage on this kind of data.