2
$\begingroup$

For personal research, I'm doing some analysis on collected data and trying to develop relationships between two variables where the data is collected through a data logger. I'm hypothesising that a = alpha * b for any point in my data logger.

Anyways I plotted the 5,000+ x,y points below, and the only relationship I see is a cloud or airplane wing shaped one where it is linear at the bottom and and a curved line on the top. Doing log scales didn't help either.

EDIT So I'm not allowed to submit image tags as a new user, but here's the graph

How can I improve my data analysis and is there a tool / cheat sheet that recommend a statistical analysis for a type of data.

It'll be nice if you can help find a solution; however, my goal is to learn about statistical analysis tools, that'll help me better understand the data and solve the problem.

Improve this question
$\endgroup$
1
  • $\begingroup$ I don't think this question is appropriate for MO as it's not a research level mathematics question. However, anyone interested in this question, please have a look at the new proposed statistics stack-exchange site. It would be great if that could get off the ground and running so that interesting (but inappropriate for MO) questions like this one can find a home. area51.stackexchange.com/proposals/33/statistical-analysis $\endgroup$
    – Noah Snyder
    Commented Jun 18, 2010 at 0:01

5 Answers 5

Reset to default
2
$\begingroup$

To find the relationship between the data try this tool - to all of my knowledge this is the best one available (at least I am very excited about it)

http://ccsl.mae.cornell.edu/eureqa

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ Eureqa is amazing $\endgroup$
    – dassouki
    Commented Jul 8, 2010 at 17:41
1
$\begingroup$

I am not sure what exactly you are looking for. But, looking at the data, a linear one dimensional model does not fit so well. To see this, try CCA (Canonical Correlation Analysis): the linear version computes two linear transforms of the input spaces, such that the transformed data are maximally correlated. I suspect from your data that the maximum correlation you will get is no more that 0.7-0.8. If you are looking to "explain the data", you need a generative model, perhaps a Gaussian Mixture for good results.

For the practical aspects of data analysis, you can try Handbook of Statistical Analysis and Data Mining Applications by Nisbet, Elder, Miner. If you look to understand the theory, The Elements of Statistical Learning by Hastie, Tibshirani and Friedman is the standard graduate text (though it might be too difficult, if you are not at ease with probability, measure theory and some functional analysis.)

Improve this answer
$\endgroup$
0
1
$\begingroup$

Besides log-transforming the y scale, as proposed, you could try a kernel regression, maybe a kernel regression taking into account the highly skewed nature of y. ¿Why not post the data here, so we have something to play around with?

Improve this answer
$\endgroup$
1
  • $\begingroup$ Looking at the data again , it may be that a two-dim kernel density estimate could be more usefull than a kernel regression. As always for data analysis questions, we need more of the applied background. $\endgroup$
    – kjetil b halvorsen
    Commented Jun 18, 2010 at 22:34
1
$\begingroup$

For analyzing data using a computational programing language the "R Graphs Cookbook" would be a great buy, especially if one has some previous experience with R Programming Language.

Improve this answer
$\endgroup$
0
$\begingroup$

The first thing I'd do if I saw a graph that looked like that is replace the y-value with the logarithm of the y-value and see what that looks like.

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .