Questions tagged [data-analysis]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
37 votes
5 answers
7k views

Inference using Topological Data Analysis: Is it worth it for a regular statistician to learn TDA?

After having read Gunnar Carlsson's Topology and Data I feel enthusiastic to use some topological data analysis (TDA) methods in my current research, mostly in social sciences. We often handle huge ...
Mauricio Tec's user avatar
21 votes
5 answers
3k views

Easier reference for material like Diaconis's "Group representations in probability and statistics"

I'm teaching a class on the representation theory of finite groups at the advanced undergrad level. One of the things I'd like to talk about, or possibly have a student do any independent project on ...
Ben Webster's user avatar
  • 43.9k
14 votes
4 answers
736 views

Category of data sets, motivated by persistent homology?

Is there a useful or agreed-upon category of data sets? In particular, I'm thinking about a point cloud and wondering what an acceptable morphism between point clouds "should" be. Edit/Clarification:...
cheyne's user avatar
  • 1,396
11 votes
3 answers
2k views

Higher categories as data structures

Still wading through higher category theory. I find the subject a bit intimidating, not so much for technical reasons, but because I lack sufficient intuition as to the motivation(s)/heuristics one ...
Mirco A. Mannucci's user avatar
7 votes
3 answers
54k views

Correlation between 3 variables

For correlation measurement betweeen 2 variables, I use Pearson formula. What formula can use to find degree of correlation between 3 variables ? My variabes are not symmetric: The correlation in ...
Andrei's user avatar
  • 195
6 votes
1 answer
628 views

Can you do geometry with persistent homology?

Setup In practice, persistent homology of data $X$ is often used to infer the homology of the underlying (Riemannian) manifold $M$ that the data is sampled from. However most filtrations (Vietoris, ...
Alex's user avatar
  • 139
6 votes
1 answer
327 views

Approximate homology of a large simplicial complex

I can use software to calculate the Betti numbers $\beta_0,\beta_1,\beta_2,\dots$ of a finite simplicial complex. This is prohibitive for large complexes, built on say > 100,000 nodes. Is there some ...
apg's user avatar
  • 612
5 votes
2 answers
334 views

Is there an upper bound on the number of points in point cloud for which we compute the persistent homology?

I am interested in the topic of persistent homology (topological data analysis). According to what I read, there is some roadblock in the analysis of "big data" using persistent homology as it is ...
yoyostein's user avatar
  • 1,219
5 votes
1 answer
2k views

t-Stochastic Neighbor Embedding vs Topological Data Analysis

The shortest form of this question is: How much TDA can be done with tSNE? Specifically, I'm referring to the application of TDA to clustering data, so, think along the lines of Ayasdi's ...
Alex R.'s user avatar
  • 4,902
5 votes
2 answers
2k views

Solving for Moore Penrose pseudo inverse

I have a system to solve, set up as : $$Ax = b$$ with a square rank deficient matrix $A$. The paper suggests to use a Moore Penrose pseudo inverse, which in my case can be computed using the ...
WhitAngl's user avatar
  • 481
5 votes
0 answers
188 views

Gaussian curvature/Euler characteristic of Facebook clusters

If I look at a connected subgraph on a small collection of actors (such as a small cluster) in the Facebook social network, and I find that 1) The Euler characteristic of the clique complex built on ...
apg's user avatar
  • 612
5 votes
0 answers
2k views

Find the axis of symmetry in a point cloud

I have some dataset which describes a spherical cloud of points in 4D space. Actually, the coordinates of the points are the coefficients of unit quaternions, so you get the idea on what the data is ...
noncom's user avatar
  • 151
4 votes
1 answer
1k views

How to measure distribution of high-dimensional data

I have to methods of projecting random samples in $\mathbb{R}^n$ onto a manifold defined by $C(q)=0$, which is a lower-dimensional subset. Now, samples in $\mathbb{R}^n$ are uniformly distributed. ...
Jeremy Span's user avatar
4 votes
1 answer
256 views

Advantage of fractional Fourier transform over multiscale wavelet

What is the best argument of fractional Fourier transform over multiscale wavelet in data analysis purpose. Optimization of the good time-frequency domain parameter ? "Good" will be, find the %time-%...
sharl's user avatar
  • 41
4 votes
0 answers
355 views

Persistent homotopy groups

Everybody in algebraic topology loves homology and cohomology, but sometimes we like homotopy groups also, since they detect different things (think about spheres) . An interesting and recent ...
Andrea Marino's user avatar
4 votes
0 answers
498 views

Implications of a recent result on Benford's law

I want to the discuss the implications of a theorem by J. Morrow (2010) regarding Benford's law. There are many papers written about Benford's law with a comprenhensive discussion of the advantages ...
Mauricio Tec's user avatar
3 votes
1 answer
723 views

name for $\underset{x}{\operatorname{argmin}} \displaystyle\sum\limits_{i=1}^n |x_i - x|$

Given a real-valued data set $ x_1, \dots, x_n $, what do you call the quantity $$\underset{x}{\operatorname{argmin}} \displaystyle\sum\limits_{i=1}^n |x_i - x|$$ This seems like a pretty basic ...
Diogenes Creosote's user avatar
3 votes
1 answer
200 views

Why do we consider some weakening frames like K-frames, frame sequences, and upper semi-frames?

I have found some applications of the Frame Theory in engineering sciences like signal processing, image processing, data compression, sampling theory, optics, filter-banks, signal detection. As we ...
ABB's user avatar
  • 3,898
3 votes
1 answer
275 views

Topology data analysis - faster algorithm

The Topology Data Analysis uses the Mapper algorithm, but computational complexity is not good. Is there an alternative algorithm for algorithm Mapper? Is there an algorithm that works faster?
Karolina Paradysz's user avatar
3 votes
2 answers
144 views

Checking $f(x_1,y_1)f(x_2,y_2)-f(x_1,y_2)f(x_2,y_1) \ge 0$

I am working in data science and I have to deal with the following problem for which I would like to find a simplification: We call a function almost positive if $f(x_1,y_1)f(x_2,y_2)-f(x_1,y_2)f(x_2,...
Xing Wang's user avatar
  • 119
3 votes
1 answer
64 views

Are Optimal Tours Sensitive to Clusters?

Background of this question is that I had been asked for advice in clustering a very big set ($10^6$ to $10^8$) set of points in Euclidean 3D-space; these points in turn lie on 2D manifolds. I ...
Manfred Weis's user avatar
  • 12.6k
2 votes
1 answer
469 views

Latent Dirichlet allocation - math words digest ?

Latent Dirichlet allocation - is quite a popular topic in data-mining. Wikepedia mentions thousands citations in few years. Question 0 Can one give some digest for a math minded person of the key ...
Alexander Chervov's user avatar
2 votes
1 answer
180 views

On the entries of a matrix representation for a boundary operator of a persistence module

In equation 6 of Computing Persistent Homology (page 8), the authors put forward the following identity: $$\deg \hat{e_i}+\deg M_k (i,j)=\deg e_j$$ Where $\hat{e_i}$ and $e_j$ are elements of ...
Eben Kadile's user avatar
2 votes
1 answer
105 views

Compact objects in persistence modules and interval decomposition

$\newcommand\Mod{\mathrm{Mod}}\DeclareMathOperator\Fun{Fun}$If $k$ is a field, a persistent $k$-module is a functor $\mathbb{R}\to \Mod_k$ where $\mathbb{R}$ is a poset under the natural ordering of $\...
dicemaster666's user avatar
2 votes
1 answer
115 views

Quantifying an increasing spacing between data points

Is there a measure or statistic that could quantify a steady increase in the spacing between data points in a time series? For instance, in the figure, the points are clustered and dense near 0, but ...
Raskol's user avatar
  • 167
2 votes
1 answer
134 views

Solution of the k-means problem in a simple case

Consider the setup of the $k$-means problem and assume that the data points are confined to $k$ balls of radius $\varepsilon$ while the pairwise distances between the centers of the balls are $> 2 \...
passerby51's user avatar
  • 1,639
2 votes
2 answers
1k views

Appropriate histogram comparison distance measure

I am working with hyperspectral image data in R, so I have subset an image to a region of 5000 pixels, each containing a vector 254 bands in length. I would like to cluster this data in order to try ...
EJA's user avatar
  • 23
2 votes
1 answer
96 views

PCA-like method for filtering known variances

Principal Component Analysis is used to reduced the dimensions of atmospheric pressure grids (lat X long X time) into their most important modes of behaviour (e.g, the North Atlantic Oscillation is ...
Will Rust's user avatar
2 votes
0 answers
42 views

Robustness of largest singular vectors with respect to noise

I would like to find a result that shows that the largest right-singular vectors of a data matrix are in some sense robust with respect to low-variance noise perturbations. Specifically, let $X = U D ...
foobar_98's user avatar
2 votes
0 answers
181 views

Discrete Morse theory, choice of Morse function, and removing noise

If I have a simplicial complex, and a discrete Morse function defined on the simplices, I can use persistent homology to produce a barcode which helps me distinguish "persistent" shape from noise. To ...
apg's user avatar
  • 612
2 votes
1 answer
584 views

correlation for three variables? [closed]

suppose we have three variables here, x,y, z now, what we know is that the correlation between x and z is 0.6, the correlation between y and z is 0.65. Here is the question, is there any formula to ...
user33817's user avatar
2 votes
0 answers
201 views

isomap and self intersections

I sample a 2D surface in $\mathbb{R}^3$ with $N$ points, and compute an isomap using pairwise weighted geodesic distances. I am thus able to embed this surface into a $M$ dimensional space in which ...
WhitAngl's user avatar
  • 481
2 votes
0 answers
858 views

Classical Multidimensional Scaling

Hi, I am doing an MDS with a distance matrix coming from geodesic distances between points X on a 3d mesh (ie., not euclidean distances), and try to find points Y in euclidean space which best ...
WhitAngl's user avatar
  • 481
1 vote
2 answers
1k views

Interpolating a "manifold" between two points

Edit: I have reworded the question. This may be a basic question but I am having trouble figuring out the correct answer. I want to find a local coordinate chart that fits a d-dimensional ...
Chirag Lakhani's user avatar
1 vote
2 answers
2k views

What is the uncertainty on the (Pearson) correlation coefficient?

Do you know what is the uncertainty on the Pearson correlation coefficient as a function of the uncertainty on the measurement in the data set. I know of an expression giving the uncertainty related ...
user655870's user avatar
1 vote
1 answer
60 views

Performing Statistical Analysis on a Data Set With a lot of Null Responses

I am currently trying to perform some statistical analysis on some data to see if there is any meaningful conclusion for a research project I am working on; however, I have come across a problem. ...
Stephen Fratamico's user avatar
1 vote
1 answer
134 views

Two theorems about incoherence

These are two theorems I have heard being referred to in "folklore" but I cant find the proofs for these in any compressed sensing or high-dimensional probability reviews (like, https://www.math.uci....
gradstudent's user avatar
  • 2,136
1 vote
1 answer
93 views

Morlet wavelet transform of binary dataset in R

I want to perform a Morlet Wavelet transform analysis (WTA) on a sequence of binary data (0, 1), length about 19000 observations. The result seems reasonable, but I have my doubts whether WTA can be ...
Istvan Gabor Hatvani's user avatar
1 vote
1 answer
209 views

Invariants ("checksums", "hash") for collection of integers

The sum of a collection of integers doesn't depend on the order of the integers and can detect the corruption of one element of the collection (but multiple elements can get corrupted without their ...
user1823664's user avatar
1 vote
1 answer
187 views

What subjects of Fourier analysis have had more effect on machine learning? [closed]

What is the salient uses of Fourier analysis in machine learning? What subjects of Fourier analysis have had more effect on machine learning? Please mention the references.
ABB's user avatar
  • 3,898
1 vote
0 answers
39 views

Multidimensional scaling with partially known distance matrix

As far as I know, multidimensional scaling requires a matrix of pairwise distances between the data points to be available. What if I only have distances between some pairs of points, but not all of ...
user3749105's user avatar
1 vote
0 answers
65 views

Find Kullback-Leibler distance between two densities [closed]

can someone help me with this exercise? (look at the image). How can I find Kullback-Leibler distance between this two densitie? I have no idea how to arrive at the solution. Every suggestion is ...
Empty's user avatar
  • 11
1 vote
0 answers
128 views

Implementation of Mellin transform of exponential decay

I'm trying to understand this paper: 10.1016/j.jmr.2010.05.015. It is about using a Mellin transform of curves that contain multiple exponential decays of varying contributions (CPMG data from Nuclear ...
K.Cl's user avatar
  • 111
1 vote
1 answer
47 views

Represent multivariate data [closed]

I am not sure if this is the best place for my question. Please delete if it is not, but I would really appreciate some suggestions. I want to graphically represent multivariate data. I have 7 ...
Jonathan F's user avatar
1 vote
0 answers
78 views

Bayesian inference of stochastically evolving model parameters

I have a question related to self-calibration in radio interferometry, but I will try to phrase it as generic as possible. I have a set of data points, $D = \{ d_{0, t_0}, d_{1, t_0}, ..., d_{M, t_0}, ...
Sketos's user avatar
  • 29
1 vote
0 answers
91 views

What is intuitive perception of $T_{\alpha_1} \circ T_{\alpha_2} \circ ... \circ T_{\alpha_M} $ in graph domain?

if $G(V,E,W)$ be a weighted graph and $\vert V \vert =N $. for any vertex $i \in \lbrace 1,2,...,,N \rbrace $ define a generalized translation operator $T_i:\mathbb{R}^N \to \mathbb{R}^N $via ...
niloofar jamshidi's user avatar
1 vote
0 answers
49 views

Calculating the density of data points around a specified point in a k-dimensional space [closed]

I am looking for a way of calculating how close data points are to a specified point in a k-dimensional space. My current method involves pythag to calulate the distance between the specified point ...
ajb's user avatar
  • 11
1 vote
0 answers
48 views

How to define a harmonic coordinates on data graph?

Suppose I knew the Ricci curvature at some point of the Manifold along several directions (the number of directions should be much more than the dimension of the manifold). Can I decompose the Ricci ...
IHitSquirrel's user avatar
1 vote
0 answers
196 views

Quantification of the extent of periodicity in a time series using fractal analyses

I need metrics to quantify and compare the extent of periodicity between any two given time series, considering the time series were "almost periodic". By "almost periodic" I mean: if I were to take ...
np20's user avatar
  • 111
1 vote
0 answers
57 views

Open volumetric time series data set

Does anyone know where I can find a good open volumetric time series data set? I had a look at some of Stanford's open data sets (https://graphics.stanford.edu/data/voldata/ ) But these do not seem ...
sav's user avatar
  • 191