Kummer's remark is from his 1860 obituary of Dirichlet, so it is an early source; a recent reference that identifies the unit theorem is page 472 of "Geschichte der Universität Unter den Linden, 1810-2010: Genese der Disziplinin : die Konstitution der Universität". I made a screen shot of the text. A reference is given to Dedekind 1871. This text is online and searchable; I did not find any mention of the story there.
Franz von Krbek describes what seems to be a different story on page 11 of his 1964 book "Über Zahlen und Überzahlen", which he attributes to Marcel Riesz, stating that Dirichlet discovered his pigeonhole principle during the Easter mass in the Sixtine Chapel: Professor Dirichlet aus Berlin 1843 gerade während der Ostermesse in der sixtinischen Kapelle in Rom die Tragweite des Schufächerprinzips in der Zahlentheorie erkannte. He also writes that he attempted to find evidence for this story in Mrs. Dirichlet's correspondence to her sister, without success.
Krbek expresses doubt about the reliability of Riesz's statement, which might just be another interpretation of Kummer's 1860 remark. That seems to be the earliest source, subject to various interpretations on what the problem might have been.