Take all the $n\times n$ matrices of 0's and 1's and define an equivalence relation as follows: Two matrices are equal if there is a way to pass from the one to another by alternating the columns and the rows (acting by $S_n$ on the columns and on the rows).
Is there a good way to determine whether two such matrices are equal?
Are there any good invariants (polynomials, etc.)?
The obvious invariant is that the sum of the 1's on the rows and on the columns does not change.
This is the graph isomorphism problem.
More precisely, if you have to permute the rows and the columns by the same permutation, then this is graph isomorphism (use $1$ to code "edge present" and $0$ to code "edge absent".) If you are allowed to use different permutations on rows and columns, then this is bipartie graph isomorphism, which is equivalent to graph isomorphism.
In practice, algorithms for Graph Isomorphism are pretty fast; however, it is not known whether there is a polynomial time method to test whether two graphs are isomorphic or, equivalently, whether two matrices are equal under your operations.
The determinant would be invariant under the permutations you outlined.
Just a confirmation that the row/col sum fingerprint will not work: both matrices have the same row and column sums but are not permutations of each other.
[0, 0, 0, 1]
[0, 1, 1, 1]
[1, 1, 1, 0]
[1, 1, 1, 1]
[1, 0, 0, 0]
[0, 1, 1, 1]
[0, 1, 1, 1]
[1, 1, 1, 1]
