Inference using Topological Data Analysis: Is it worth it for a regular statistician to learn TDA?

Question

After having read Gunnar Carlsson's Topology and Data I feel enthusiastic to use some topological data analysis (TDA) methods in my current research, mostly in social sciences. We often handle huge databases and I think it can be an interesting exercise to do TDA. I wonder two things about the current state and purpose of TDA (I do understand TDA's main advantages are not for the social sciences though):

1. Are there TDA methods than can be used to establish a relationship among a variable of interest and a set of (possible) explanatory variables? That is, to do some sort of statistical inference?
2. Can we make predictions based on TDA tools?

Any bibliographical reference or explanation about why this is not possible is appreciated.

Answer 1

Let me answer the broad question first: depending on what you actually want to do, the barcode-type invariants extracted by topological data analysis could be quite useful in your work. And it doesn't take too much prior knowledge to use the TDA tools. For instance, if all you want to do is show that two datasets are qualitatively different, you can just compute their barcodes (I've written software to do this, as have others) and calculate the difference between them. It's easy, fast and free so why not try something orthogonal and complementary to your existing techniques?

---

The usual pipeline for TDA is as follows: starting with your data, you impose the structure of a filtered cell complex, compute the persistent homology, and output the barcode. The reason you may find it difficult to get precise answers to your questions is quite simple: everything depends on how the filtration is concocted! It is a bit of an art form to know exactly what to compute the persistent homology of, given the features that you actually care about.

---

Here's a typical TDA approach to your first question: let the dependent variable be $x$ and the independent variables $y_1,\ldots,y_n$.

Assume, by thresholding and binning if necessary, that each $y_j$ attains only finitely many states. Construct the flag complex on the $n$-partite graph whose vertex-bins correspond to values attained by the $y_j$s. Each simplex is weighted by the minimum $x$-value corresponding to the $y$-values fixed by its vertices. These weights give you a filtered simplicial complex, and generators of the $0$-dimensional persistence intervals of the super-levelset filtration tell you which configuration of $y$s correspond to which $x$ values.

---

Regarding your second question, a lot depends on what you mean by "predictions". For two examples of using persistent homology for predictions, consider Liz Munch's PhD thesis available here. It is possible to predict -- up to an extent -- coverage failure in sensor networks modeled as devices with a given failure probability. Perhaps a modification of this model could make predictions in situations that are of interest to you.

Answer 2

Update: There is a now a new paper by Otter et al which went through all the trouble of comparing many software packages for performance, memory, ease-of-use etc.: http://arxiv.org/abs/1506.08903

This new answer is in response to the comment below my original answer asking about persistent homology software other than phom... adding all this information into the existing answer would make things unwieldy. In any case, here is a list of available implementations that I've come across:

1. jPlex is a library written in java. It is maintained by Carlsson's CompTop group. It handles filtered simplicial complexes and outputs barcodes. This tutorial explains how to run jPlex within Matlab, but you can also use it standalone via Beanshell.
2. Dionysus is written in C++ and maintained by Morozov. On his page you can find ways to bind the code to python, but note that you will need at least Boost and CMake to get even the basic modules to compile. It handles filtered simplicial complexes of various kinds (eg Rips and Cech complexes).
3. I know very little about the awesomely-named phat C++ library except that it claims to compute persistence in a distributed way, which may be useful if you are running out of memory. The underlying paper which explains the parallelization is here.
4. Mrozek has a standalone software package written in C++ to compute persistent homology of filtered cubical grids which are input as text files.
5. There's my own Perseus which is completely standalone by design. It handles both cubical and simplicial filtrations. All input and output is handled via simple text files (see the linked page for format descriptions). It uses discrete Morse theory to drastically reduce the size of the input filtration while preserving persistent homology groups, as explained in this paper, so I've found it to be typically faster and less memory-hungry than at least jplex and dionysus.

Finally, note that the original algorithm by Zomorodian and Carlsson is not difficult to implement in your favorite language or environment. So if none of the aforementioned packages are suitable, you could write and debug your own in less than a week given moderate programming skills.

Answer 3

Topological data analysis is very much in its infancy as a field. Right now there's far more methods available than theorems and the usual "intellectual infrastructure" one might expect from a branch of mathematics/statistics.

My impression is at present people are largely playing with all the ideas in various applied situations, trying to see what kind of inferences the tools allow one to make. This is in contrast to the desire to prove fundamental theorems. Many very basic questions are still open about the tools that persistent homology and its sibling ideas present.

So to answer your questions (1) and (2), yes there are such tools. But given that the field is relatively primordial, the extent you can use the tools to do what you like are largely governed by how lucky you get and how well you can make inferences between topologial computations like homology and geometric intuition.

Answer 4

There was a workshop on topological data analysis (TDA) at the Institute for Mathematics and its Applications in October 2013.

Link: http://www.ima.umn.edu/2013-2014/W10.7-11.13/

Note in particular the tutorial on statistical inference for TDA by Alessandro Rinaldo: http://ima.umn.edu/videos/?id=2443, and see the CMU TopStat page at http://www.stat.cmu.edu/topstat/.

Update: there is an R package available for TDA - see http://www.stat.cmu.edu/~flecci/software/index.html. See also the tutorial at http://www.stat.cmu.edu/topstat/Talks/files/Jisu_150623_TDA_tutorial.pdf.

Answer 5

The standard computational package in this area has become javaPlex. It is written by a fellow at Stanford (Andrew Tausz) and is a major update to the previous jPlex. It seems to be the best in terms of being actively managed and easy to use. In addition to the library code, it contains many usage examples.

The main site for it is: http://appliedtopology.github.io/javaplex/.