Binary quadratic forms arise in nature as norm forms for a quadratic field. This point of view has various consequences in number theory.
For a fixed negative discriminant (the definite case), Gauss discovered that the quadratic forms (or indeed their $\mathrm{SL}_2(\mathbb{Z})$ equivalence classes) can be composed. This led him to the phenomenon of the ideal class group -- or Picard group, as the arithmetic geometer would have it, -- before ideals were actually invented by Kummer and Dedekind. Besides in the ideal class group for "higher"more general number fields, Gauss's composition law has also found a differentan extension in Bhargava's higher composition laws. These are based on the representation theory of arithmetic groups ($\mathrm{SL}_2(\mathbb{Z})$ and its generalizations), in which regard they are natural structures in themselves. They have striking consequences inapplications to old problems regarding mean asymptotics of Selmer ranks of elliptic curves, the $3$-parts of class groups of quadratic fields, etc.
The Epstein zeta function takes the shape $\zeta_Q(s) := \sum_{\mathbf{n} \neq \mathbf{0}} Q(\mathbf{n})^{-s}$, for a given signature $(d,0)$ quadratic form $Q$. It has all the right analytical properties (meromorphic with simple pole at $s = 1$ and a functional equation relating $s \leftrightarrow 1-s$), allowing to decompose the zeta function of an imaginary quadratic field over a set of representatives $Q$ for the class group. This has consequences for the arithmetic of these fields, beautifully developed in Siegel's Lectures on Advanced Analytic Number Theory (Tata Institute lecture series, 1961).
For $d = 2$, $\zeta_Q(s)$ is in effect an Eisenstein series ($|mz+n|^2$ being a binary quadratic form in $m,n$), which is a natural structure all over mathematics, being a continuum of modular forms forming the continuus part ofin the spectral resolution of the hyperbolic Laplacian. Siegel apparently had much interest in the conceptual role played in number theory by the higher rank quadratic forms and their Epstein zeta function. Much of his work was put on representation theoretic footing in Weil's 1964 paper Sur certains groupes d'operateurs unitaires. Michael Berg's book, The Fourier-Analytic Proof of Quadratic Reciprocity, is a terrific introduction to these ideas.