I have a following scenario. there is a huge collection of data resulting from a random experiment $E$ (I do not say random variable yet, for reasons that you will need to explain in your answer). Let us call this a collection $D$. I have partitioned this randomly into $4$ equal parts $D_1,D_2,D_3,D_4$. The data points are nothing but a $d$-dimensional vectors, where $d$ is a very large number, $d$ higher than even the size of $D$. Now I want a low dimensional representation of this data but which preserves the geometry, that is inter-point distances of all the data points. For this I do two experiments $A$ and $B$.

Experiment $A$ :

Project all the data points of $D$ onto the span of all points in one segment, say $D_1$. This method preserves the interpoint distances of all points in $D_1$ exceedingly well, but does badly when you consider all points in $D$ rather than just $D_1$. The inter point distances of points from $D_2,D_3,D_4$ are not preserved satisfactorily, which is a very understandable thing. Similar is the case if you consider span of $D_2$ or $D_3$ or $D_4$.

Experiment $B$.

What I do is, I take one segment say $D_1$, and compute Eigen vectors $e_1,e_2,..e_k$ corresponding to top $k$ Eigen values (which contain the bulk of the energy) of the correlation matrix of data in $D_1$. Here $k<<n(D_1)$. Now I project all the data in $D$ to the span of $\{e_1,e_2,e_3,..e_k\}$, and surprisingly it preserves the interpoint disntances of all points in $D$ very well, (all points in $D$ and not just in $D_1$). This preserves the distances considerably better than in the case of old experiment $A$, where we project data in $D$ to the span of all points in $D_1$.

Is there any underlying mathematics that can explain this phenoemnon? What is the mathematics of the distribution of the random experiment that is generating this data. I want to know the strange facts about the underlying distribution of the random experiment that is generating this data.

  • $\begingroup$ try en.wikipedia.org/wiki/Multidimensional_scaling $\endgroup$
    – user83457
    Jun 21 '16 at 9:24
  • $\begingroup$ We cannot model $D$ as a random variable but something more general than that. $\endgroup$
    – Rajesh D
    Jun 21 '16 at 10:48
  • $\begingroup$ @StevenLandsburg : "This preserves the distances considerably better than in the case of old experiment A", why is it better than that in experiment $A$. That is the question? $\endgroup$
    – Rajesh D
    Jun 21 '16 at 15:47
  • $\begingroup$ You are right this difference does look strange. If I had such a finding I would think the most likely answer is a programming bug or error in the data input or the random selection or whatever. Maybe you should analyse more to rule out these things. For example are the Eigenspaces of $D_2, \ldots, D_4$ really the same or close? What happens if you increase the number of Eigenvectors, i.e. if $k$ gets close to $n(D_1)$. Does the same occur if you use $D_2$ instead of $D_1$ and so on. $\endgroup$
    – g g
    Jun 21 '16 at 17:15
  • $\begingroup$ @gg : I have ruled out experimental error by repeating the experiment with various versions of data sets. Beyond a point, it get progressively worse with increase in $k$. The same if I choose $D_2$ or $D_3$ instead of $D_1$. The experiment was beaten to death to avoid any error. $\endgroup$
    – Rajesh D
    Jun 22 '16 at 2:03

Let the subspace spanned by $\{e_1,e_2,...e_k\}$ be $S_k$. the experiemnt $B$ giving good results imply, very high say 90% of energy of entire $D$ is in subspace $S_k$. Let the span of all vectors of $D_1$ be denoted as $S$. We know $S_k \subset S$, so experiment $A$ should yield same or slightly better results than experiment $B$. This contradiction shows that this scenario is very improbable and points to experimental mistakes. Please let me know if I miss anything.

PS : here I assume mean is stationary and same for $D_1,D_2,D_3,D_4$.

  • $\begingroup$ Wonder what kind of an error in measurements would cause this effect? Does that error has a statistical model? $\endgroup$
    – Rajesh D
    Jun 24 '16 at 10:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.