MathOverflow is a question and answer site for professional mathematicians. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top


Are there efficient algorithms to check if a finite simplicial complex defined in terms of its maximal facets is shellable?

By efficient here I am willing to consider anything with smaller expected complexity than the exponential mess one gets by naively testing all possible orderings of maximal facets.


Let $\Delta$ be a simplicial complex and for each simplex $\sigma \in \Delta$ let $\bar{\sigma}$ denote the subcomplex generated by $\sigma$ and all its faces. Fix an ordering of its maximal facets $F_1,\ldots,F_K$, pick some $k \in \lbrace 1,\ldots,K\rbrace$ and define $\Delta_k$ to be the subcomplex generated by $\bigcup_{1\leq j \leq k} F_j$, i.e., all facets up to and incluing the $k$-th one.

Definition: We call this ordering of maximal facets a shelling if the intersection $\overline{F_{k+1}} \cap \Delta_k$ is a simplicial complex of dimension $\dim (F_{k+1}) - 1$ for each $k \in \lbrace 1,\ldots,K-1\rbrace$.

In general, the complex $\Delta$ need not be a combinatorial manifold or have a uniform top dimension for its maximal facets. It is known that if $\Delta$ is shellable then there exists a shelling by maximal facets ordered so that the dimension is decreasing along the order. So one method to simplify the computational burden is to test only those orderings $F_1,\ldots,F_K$ of maximal facets so that $\dim F_i \geq \dim F_j$ whenever $i \leq j$, but of course in the worst case all these facets could have the same dimension.


Shellability is an extremely useful notion in topological combinatorics: many interesting simplicial complexes and posets in this field turn out to be shellable. I refer you to the works of Anders Bjorner and others for details, see here or here or... Since every shellable complex is a wedge of spheres, establishing shellability leads to all sorts of interesting conclusions. Among other things, shellable complexes must lack torsion in homology of all dimensions.

share|cite|improve this question

As you point out (relayed from Frank Lutz), it seems likely that checking shellability is NP-hard.

But all is not lost:

  1. A complex that is shellable usually has lots of shellings, and it's often quick to find them by recursively trying to extend a partial shelling. The above-mentioned answer mentions some ways that this can be made more efficient.

  2. A (pure) complex that is not shellable often has a negative component in its $h$-vector, a certain re-encoding of the $f$-vector. See
    For non-pure complexes, you can check the so-called $h$-triangle -- see Björner and Wachs, "Shellable nonpure complexes and posets I".
    Since shellable complexes (more generally sequentially Cohen-Macaulay complexes) have positive $h$-triangles, this gives a quick way of eliminating some complexes that are not shellable.

You could get a little more involved with (2), and check for positive $h$-triangles of every link in the complex, since a link in a shellable complex is shellable.

In practice, when I've had complexes that I've wanted to computationally check shellability on, I've generally found the combination of the two approaches to give me an answer. You can either first check for obvious non-shellability with (2), and if everything is positive apply (1); or else first check shellability with (1), and if the computation appears to hang, then look for a negative entry in the $h$-triangle.
(But this works for me partly because the complexes I look at usually arise from some kind of "nice" combinatorial object or process.)

share|cite|improve this answer
up vote 8 down vote accepted

Since there were no answers for a few months, I asked this question to my colleague and triangulation expert Frank Lutz. Since his response was wonderful and exhaustive, I am reproducing it here for the benefit of others who find such matters interesting.

Spoiler alert: it is very hard to test for shellability.

Testing shellability is a mess. The complexity status is open, but believed to be NP complete. There is an implementation in the polymake package:

--> Follow the links:

    + SimplicialComplex
       + Combinatorics
           + Shellable

As far as I remember, the polymake implementation is based on the Shellability checker of Masahiro Hachimori. The bad news is that the procedure uses backtracking/reverse search and thus basically goes through all possible permutations. The good news is that local conditions on homology vectors are checked, which allows one to cut the search tree and speed up computations.

In the special case of triangulated 3-balls/3-spheres, I wrote backtracking code for testing shellability myself:

 # vertices     # triangulated 3-balls   # non-shellable 3-balls
      4                     1                       -
      5                     3                       -
      6                    12                       -
      7                   167                       -
      8                 10211                       -
      9               2451305                      29
     10            1831363502                  277479

See and

I had a look at the 29 non-shellable 9-vertex 3-balls, the smallest of these is described here. Among these, there are rather different types. In particular, for being "shellable" or "non-shellable" it really can matter in which way some tetrahedron is attached to what has been built before, which makes testing difficult.

My guess would be that there is no way arround backtracking, although local conditions can help to achieve some speed up. For testing explicit examples (of dimension at least 3), the following will happen:

- If some example is shellable, there is a good chance
  to find a shelling sequence with backtracking.

- If some example is non-shellable (and has, say, 20 or more vertices),
  it will be hopeless to complete the backtrack search.
share|cite|improve this answer

You might also be interested in vertex decomposability; a relevant package written by David Cook (for Macaulay 2) is described in this paper:

share|cite|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.