Here's a translation of part of the address dealing with the Fifth Problem (source):
For infinite groups the investigation of the corresponding question is, I believe, also of interest. Moreover we are thus led to the wide and interesting field of functional equations which have been heretofore investigated usually only under the assumption of the differentiability of the functions involved. In particular the functional equations treated by Abel$^{12}$ with so much ingenuity, the difference equations, and other equations occurring in the literature of mathematics, do not directly involve anything which necessitates the requirement of the differentiability of the accompanying functions. In the search for certain existence proofs in the calculus of variations I came directly upon the problem: To prove the differentiability of the function under consideration from the existence of a difference equation. In all these cases, then, the problem arises: In how far are the assertions which we can make in the case of differentiable functions true under proper modifications without this assumption?
The paper you link to deals with the functional equation of Abel dealt with in the paper
- N.H. Abel, Ueber die functionen welche der Gleichung $\varphi x + \varphi y = \psi(xfy+yfx)$ genugthun, Journal für die reine und angewandte Mathematik, herausgegeben von Crelle, Bd. 2, Berlin 1827. (DigiZeit) (French translation Sur les fonctions qui satisfont à l'équation $\varphi x + \varphi y = \psi(xfy+yfx)$ available in Oeuvres complètes de Niels Henrik Abel pp 389–398, available at the Internet Archive).
The famous Cauchy functional equation $f(x+y) = f(x)+f(y)$ is a very special case, and is a baby case of the usual, finite-dimensional version of Hilbert's Fifth Problem. Or, one can view Abel's functional equation as a variant of Cauchy's with an 'exotic' addition, so in some sense a non-standard group operation on the reals. One can interpret, I gather, the group laws for a (local?) Lie group as a bunch of (coordinate) functions obeying a functional equation. Imagine you had no concept of manifold, topological space etc, but could write down what associativity meant direction in terms of coordinate functions, a priori only continuous. I would tentatively guess this is what Hilbert was driving at, allowing in the remarks above something from calculus of variations not obviously coming from groups.