3
$\begingroup$

I have a following scenario. there is a huge collection of data resulting from a random experiment $E$ (I do not say random variable yet, for reasons that you will need to explain in your answer). Let us call this a collection $D$. I have partitioned this randomly into $4$ equal parts $D_1,D_2,D_3,D_4$. The data points are nothing but a $d$-dimensional vectors, where $d$ is a very large number, $d$ higher than even the size of $D$. Now I want a low dimensional representation of this data but which preserves the geometry, that is inter-point distances of all the data points. For this I do two experiments $A$ and $B$.

Experiment $A$ :

Project all the data points of $D$ onto the span of all points in one segment, say $D_1$. This method preserves the interpoint distances of all points in $D_1$ exceedingly well, but does badly when you consider all points in $D$ rather than just $D_1$. The inter point distances of points from $D_2,D_3,D_4$ are not preserved satisfactorily, which is a very understandable thing. Similar is the case if you consider span of $D_2$ or $D_3$ or $D_4$.

Experiment $B$.

What I do is, I take one segment say $D_1$, and compute Eigen vectors $e_1,e_2,..e_k$ corresponding to top $k$ Eigen values (which contain the bulk of the energy) of the correlation matrix of data in $D_1$. Here $k<<n(D_1)$. Now I project all the data in $D$ to the span of $\{e_1,e_2,e_3,..e_k\}$, and surprisingly it preserves the interpoint disntances of all points in $D$ very well, (all points in $D$ and not just in $D_1$). This preserves the distances considerably better than in the case of old experiment $A$, where we project data in $D$ to the span of all points in $D_1$.

Is there any underlying mathematics that can explain this phenoemnon? What is the mathematics of the distribution of the random experiment that is generating this data. I want to know the strange facts about the underlying distribution of the random experiment that is generating this data.

Improve this question
$\endgroup$
5
  • $\begingroup$ try en.wikipedia.org/wiki/Multidimensional_scaling $\endgroup$
    – user83457
    Commented Jun 21, 2016 at 9:24
  • $\begingroup$ We cannot model $D$ as a random variable but something more general than that. $\endgroup$
    – Rajesh D
    Commented Jun 21, 2016 at 10:48
  • $\begingroup$ @StevenLandsburg : "This preserves the distances considerably better than in the case of old experiment A", why is it better than that in experiment $A$. That is the question? $\endgroup$
    – Rajesh D
    Commented Jun 21, 2016 at 15:47
  • $\begingroup$ You are right this difference does look strange. If I had such a finding I would think the most likely answer is a programming bug or error in the data input or the random selection or whatever. Maybe you should analyse more to rule out these things. For example are the Eigenspaces of $D_2, \ldots, D_4$ really the same or close? What happens if you increase the number of Eigenvectors, i.e. if $k$ gets close to $n(D_1)$. Does the same occur if you use $D_2$ instead of $D_1$ and so on. $\endgroup$
    – g g
    Commented Jun 21, 2016 at 17:15
  • $\begingroup$ @gg : I have ruled out experimental error by repeating the experiment with various versions of data sets. Beyond a point, it get progressively worse with increase in $k$. The same if I choose $D_2$ or $D_3$ instead of $D_1$. The experiment was beaten to death to avoid any error. $\endgroup$
    – Rajesh D
    Commented Jun 22, 2016 at 2:03

1 Answer 1

Reset to default
1
$\begingroup$

Let the subspace spanned by $\{e_1,e_2,...e_k\}$ be $S_k$. the experiemnt $B$ giving good results imply, very high say 90% of energy of entire $D$ is in subspace $S_k$. Let the span of all vectors of $D_1$ be denoted as $S$. We know $S_k \subset S$, so experiment $A$ should yield same or slightly better results than experiment $B$. This contradiction shows that this scenario is very improbable and points to experimental mistakes. Please let me know if I miss anything.

PS : here I assume mean is stationary and same for $D_1,D_2,D_3,D_4$.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Wonder what kind of an error in measurements would cause this effect? Does that error has a statistical model? $\endgroup$
    – Rajesh D
    Commented Jun 24, 2016 at 10:14

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .