Continuing what was said by @GerryMyerson, the project of providing foundations for mathematics started by Frege was presented in a treatise called Grundgesetze der Arithmetik (Basic laws of arithmetic). The axioms of this treatise were proven inconsistent by Bertrand Russell in what we know today as Russell's paradox.
This paradox also affects naive set theory, understood as the theory comprising the following two axioms:
Axiom of extensionality: $(x \in a \leftrightarrow x \in b) \rightarrow a = b$. That is, if two sets $a$ and $b$ have the same elements, then they're the same set.
Axiom (scheme) of unrestricted comprehension: $x \in a \leftrightarrow$ P$x$, for each formula P$x$. That is, to each property P uniquely corresponds one set $a$.
Naive set theory, thus understood, follows from Frege's axioms and seems to capture very well the notion of set. But since $x \notin x$ is a formula, the axiom scheme of unrestricted comprehension guarantees that the following is an axiom:
- $x \in x \leftrightarrow x \notin x$
Now, when we ask whether $x \in x$ or $x \notin x$, we obtain contradictory situations in both cases.
This paradox was solved by discarding this axiomatisation of set theory and, hence, Frege's axiomatics. But some logicians, mathematicians and philosophers have considered that perhaps this wasn't the right way to solve this. Instead of rejecting this naive set theory or Frege's theory, they propose to reject the principle of explosion or ex contradictione sequitor quodlibet:
- $P \wedge \neg P \rightarrow Q$. That is, from a contradiction follows any formula or statement.
This research programme is often known as the paraconsistent programme, because they work with paraconsistent logics. A logic system is said to be paraconsistent iff the logical thesis (4) is not valid in general. Hence, if the theory is inconsistent, it doesn't mean that anything follows from it (which means that it still may be useful). You can find out more about his programme in:
You will find there (specially in the second link) a whole programme for researching inconsistent mathematical theories, which are generally considered of no mathematical interest. (You will also find that mainly philosophers are working in this programme.)
Whether this programme is of any scientific value, that's for you to judge. But I accept this probably wasn't the kind of answer you were looking for. There is a chance, however, that you find it very interesting. I hope it helps in any case.