"A History of Mechanics" by Rene Dugas is highly recommended reference, which includes discussion of Hamilton's work and its development.

Dugas writes that Hamilton's geometric optics was a new method of formalizing the collection of results that had already been obtained, and capable of being interpreted in terms of wave propagation (in Huyghens' sense) and corpuscles (in the sense of the dynamical principle of least action). Hamilton says he was "struck by the imperfection of deductive mathematical optics", and wished to give the theory the same "beauty, power, and harmony" with which Lagrange had been able to endow mechanics. [Roughly quoting from Dugas, pp.390].

The principle of least action, as arising in the works of Fermat, Maupertuis, and criticized in Euler, Lagrange, was a great controversy. An important application was Hamilton's finding the proper stationary form, and Hamilton's principle of stationary action had less contentious metaphysical interpretations.

As to practical applications, Hamilton was an astronomer, and his first (1833) article on dynamics appears to be "On a General Method of Expressing the Paths of Light, & of the Planets, by the Coefficients of a Characteristic Function". [see google books preview]. In the "Second Essay on a General Method in Dynamics", Hamilton establishes the canonical equations of motion in terms of the partial derivatives of $H$.

Jacobi appears to have simplified several redundancies in Hamilton's formalism, and is credited with geometrising the principle, in which case the characteristic function (Hamiltonian) has trajectories corresponding to the same total energy.

"A History of Mechanics" by Rene Dugas is highly recommended reference, which includes discussion of Hamilton's work and its development. Below I will summarize some interesting findings of Dugas.

(Edited Jan 13, 2021)

Dugas writes that Hamilton's geometric optics was a new method of formalizing the collection of results that had already been obtained, and capable of being interpreted in terms of either wave propagation (in Huyghens' sense) or corpuscles (in the sense of the dynamical principle of least action).

"Whether we adopt the Newtonian or Huyghenian, or any other lines of luminous or visual communication, we may regard these laws themselves, and the properties and relations of these linear paths of light, as an important separate study, and as constituting a separate science, called often mathematical optics". [Dugas, pp.391]

"But although the law of least action has thus attained a rank among the higest theorems of physics, yet its pretensions to a cosmological necessity, on the grounds of economy in the universe, are now generally rejected. And the rejection appears just, for this, among other reasons, that the quantity pretended to be economized is in fact often lavishly expended."

The principle of least action, as arising in the works of Fermat, Maupertuis, and criticized in Euler, Lagrange, was a great controversy. Hamilton found the stationary action, and thus recovered Fermat's law of least time in an acceptable variational setting.

Hamilton was astronomer who also apparently understood the calculus of variations. In his first article on dynamics "On a General Method of Expressing the Paths of Light, & of the Planets, by the Coefficients of a Characteristic Function" is included an exact statement of Hamiltonian principle of optical stationary action:

"The optical quantity called action, for any luminous path having $i$ sudden branches by reflexion or refraction, and having therefore $i+1$ separate branches, is the sum of $i+1$ separate integrals $$ACTION=V=\sum \int dV_i=V_1+\cdots +V_{i+1}$$ of which each is determined by an equation of the form $$V_i=\int dV_i=\int v_i \sqrt{dx_i^2+dy_i^2+dz_i^2},$$ the quantity $v_i$ being dependant on material properties of medium $m_i$, e.g. refractive index. This quantity $V$ is stationary in the propagation of light."

Hamilton also formulated exactly a law of varying action, which allows the ends of the optical (luminous) path to vary. He observed that the wave surfaces satisfy the equation $V=const$ and therefore the components $\partial V/\partial x$, $\partial V/\partial y$, $\partial V,\partial z$ are components of normal slowness. And redicovered the theorem of Huyghens (then disputed against the Newtonian corpuscular model) that the rays of a homogeneous system, starting from a single point or normal to a surface, remain normal to a family of surfaces after they have been subjected to any number of reflections or refractions.

Hamilton recovered Fermat's principle of least time via $$V=\int_{x',y',z'}^{x,y,z} \frac{ds}{u},$$ and $$\delta V=\delta \int v ds=0$$ corresponding to the extremal $\delta \int ds/u=0$ where $u$ is the wave (undulatory) velocity.

The above first article is on optics and variations of the optic action $V$.

In his "first essay on a general method in dynamics" he studies the living force action $V=\int_0^t 2T dt$, which is the living force (kinetic energy) accumulated from the origin of time to the time $t$.

He writes $T=U+H$, where $U=\sum m m' f (r)$ is the force between masses $m, m'$, and $$\sum m(x''\delta x + y'' \delta y + z'' \delta z)=\delta U.$$ Here $H$ is that so-called Hamiltonian term.

There's much more to say about Hamilton's work, and Dugas explains in detail.

Jacobi gave lectures in 1842-3 at Konigsberg, "Vorlesungen uber Dynamik" edited by Clebsch (1866) where Hamilton's theory is extended and simplified. In those same lectures Jacobi is credited with geometrising the principle, in which case the characteristic function (Hamiltonian) has trajectories corresponding to the same total energy.

Historical References:

"A History of Mechanics" by Rene Dugas is highly recommended reference, which includes discussion of Hamilton's work and its development.

Dugas writes that Hamilton's geometric optics was a new method of formalizing the collection of results that had already been obtained, and capable of being interpreted in terms of wave propagation (in Huyghens' sense) and corpuscles (in the sense of the dynamical principle of least action). Hamilton says he was "struck by the imperfection of deductive mathematical optics", and wished to give the theory the same "beauty, power, and harmony" with which Lagrange had been able to endow mechanics. [Roughly quoting from Dugas, pp.390].

After Fermat, Maupertuis, Euler, Lagrange, the principle of least action was still quite controversial. So Hamilton's principle of stationary action developed, which was less metaphysically contentious.

As to practical applications, Hamilton's early work was focused on astronomy, e.g. Hamilton's (1833) first article on dynamics appears to be "On a General Method of Expressing the Paths of Light, & of the Planets, by the Coefficients of a Characteristic Function". [google books gives preview]

  In the "Second Essay on a General Method in Dynamics", Hamilton establishes the canonical equations of motion in terms of the partial derivatives of $H$.

Jacobi appears to have greatly simplified Hamilton's formalism, and is credited with geometrising the principle, in which case the characteristic function (Hamiltonian) has trajectories corresponding to the same total energy.

Dugas writes that after Jacobi, further contributions were made by Liouville (1856), Lipschitz (1871), Thomson and Tait (1879), Levi-Civita (1896), and Darboux (last two chapters of his "Lecons sur la theorie generale des surfaces").

Indeed Levi-Civita's excellent text "The Absolute Differential Calculus" certainly contains discussion of geometric optics, and Hamilton's principle. See Part III, Chapter XI, pp. 287.

Historical References:

"A History of Mechanics" by Rene Dugas is highly recommended reference, which includes discussion of Hamilton's work and its development.

Dugas writes that Hamilton's geometric optics was a new method of formalizing the collection of results that had already been obtained, and capable of being interpreted in terms of wave propagation (in Huyghens' sense) and corpuscles (in the sense of the dynamical principle of least action). Hamilton says he was "struck by the imperfection of deductive mathematical optics", and wished to give the theory the same "beauty, power, and harmony" with which Lagrange had been able to endow mechanics. [Roughly quoting from Dugas, pp.390].

The principle of least action, as arising in the works of Fermat, Maupertuis, and criticized in Euler, Lagrange, was a great controversy. An important application was Hamilton's finding the proper stationary form, and Hamilton's principle of stationary action had less contentious metaphysical interpretations.

As to practical applications, Hamilton was an astronomer, and his first (1833) article on dynamics appears to be "On a General Method of Expressing the Paths of Light, & of the Planets, by the Coefficients of a Characteristic Function". [see google books preview]. In the "Second Essay on a General Method in Dynamics", Hamilton establishes the canonical equations of motion in terms of the partial derivatives of $H$.

Jacobi appears to have simplified several redundancies in Hamilton's formalism, and is credited with geometrising the principle, in which case the characteristic function (Hamiltonian) has trajectories corresponding to the same total energy.

Dugas writes that after Jacobi, further contributions were made by Liouville (1856), Lipschitz (1871), Thomson and Tait (1879), Levi-Civita (1896), and Darboux (last two chapters of his "Lecons sur la theorie generale des surfaces").

Levi-Civita's excellent text "The Absolute Differential Calculus" contains discussion of geometric optics, Hamilton's principle, and describes how Einstein's general relativity attempts to generalize Hamilton's optics. See Part III, Chapter XI, pp. 287.

I might even say the applications of Hamilton-Jacobi equations to classical mechanics continues even today(!) in the form of Monge-Kantorovich (Alexandrov-Brenier-McCann-etc.) optimal transportation. Also interesting is Nassif Ghoussoub's theory of self-dual variational problems, which is based on the classic correspondance of Lagrangian and Hamiltonian via the Legendre-Fenchel transform.