Yudkowsky and Herreshoff have a (messy but) great paper which relates the second incompleteness theorem to issues in theoretical artificial intelligence. (This paperThis paper of mine might be a more accessible introduction to the subject.) In principle, one way an intelligent agent $M$ might achieve a goal is by building an auxiliary agent $M'$ and tasking it with the goal. But presumably $M$ cannot satisfy its criterion for action unless it can prove that $M'$ reasons consistently --- otherwise it could be building an agent who might fail because it reasons incorrectly. But by the second incompleteness theorem, $M$ cannot prove that the system within which it itself reasons is sound, which means that $M'$ would have to reason within a weaker system.
It's especially a problem for the idea of self-modifying AI. A sufficiently advanced AI ought to be better at designing AI's than we are. So we might want to design an AI which is capable of improving itself by modifying its own source code. But the incompleteness obstacle seems to imply that it could only do this at the cost of weakening the formal system in which it reasons. Since proof-theoretic strength is gauged by ordinals, after finitely many iterations it would reach imbecility.
At first sight it seems like there should be a trivial resolution, but the more you think about it, the more serious you realize the problem is.
