The input of my problem is a 0/1 matrix. The problem consists in finding the largest triangular submatrix.
In my problem, a square matrix is called triangular if all the entries below (or above) the main diagonal (or the off diagonal) are zero and all the entries on the diagonal are equal to one.
For example, in the following matrix $$M=\begin{pmatrix} 1&1&1&1&1&1\\1&0&1&1&1&1\\1&1&1&0&0&1\\0&0&1&0&1&1\\1&1&1&1&1&1\\1&0&1&0&0&1 \end{pmatrix}, $$ there is an obvious 3x3 triangular submatrix $M([3,4,5],[4,5,6])$ but there also is a less obvious 4x4 triangular submatrix $M([2,3,6,4],[4,2,1,5])$.
For a given matrix, is there a known algorithm to find the largest triangular submatrix?
Thank you!