Kummer's remark is from his 1860 obituary of Dirichlet, so it is an early source; a recent reference that identifies the unit theorem is page 472 of "Geschichte der Universität Unter den Linden, 1810-2010: Genese der Disziplinin : die Konstitution der Universität". I made a screen shotscreen shot of the text. A reference is given to Dedekind 1871. This text is online and searchable; I did not find any mention of the story there.
Franz von Krbek describes what seems to be a different story on page 11 of his 1964 book "Über Zahlen und Überzahlen", which he attributes to Frigyes [Friedrich] Riesz, stating that Dirichlet discovered his pigeonhole principle during the Easter mass in the Sixtine Chapel: Professor Dirichlet aus Berlin 1843 gerade während der Ostermesse in der sixtinischen Kapelle in Rom die Tragweite des Schufächerprinzips in der Zahlentheorie erkannte. He also writes that he attempted to find evidence for this story in Mrs. Dirichlet's correspondence to her sister, without success.
Krbek expresses doubt about the reliability of Riesz's statement, which might just be another interpretation of Kummer's 1860 remark. That seems to be the earliest source, subject to various interpretations on what the problem might have been.
For the record, here is the full quote from Krbek, with translation:
In der Antrittsrede des ungarischen Mathematikers Friedrich Riesz in Szeged aus dem Jahr 1925 heisst es, dass der Professor Dirichlet aus Berlin 1843 während der Ostermesse in der sixtinischen Kapelle in Rom die Tragweite des Schubfächerprinzips in der Zahlentheorie erkannte. Woher Riesz das hat, weiss ich nicht. In den ausführlichen Briefen der Frau Dirichlet an die Schwester steht lediglich, dass, sobald die Kirchenmusik aussetzte, ein störendes Hustenkonzert einsetzte.
In the inaugural lecture of the Hungarian mathematician Friedrich Riesz in Szeged from the year 1925 it is written, that Professor Dirichlet from Berlin realized, during the 1843 Easter mass in the Sistine chapel in Rom, the range [of applicability] of the pigeonhole principle in number theory. I do not know from where Riesz got this. In the extensive correspondence of Mrs. Dirichlet to her sister it only says that, after the church music stopped, an annoying coughing concert started.