Indeed, the infimum is always zero. It's probably easier to explain this by drawing a picture to show this rather than giving a totally rigorous proof, but I'll explain how to turn the picture into a proof at the end.
We start by considering a one-dimensional interval $[0,L]$. The Neumann eigenfunctions are then given by $v_k = \cos \left(\frac{k \pi x}{L}\right) $ with eigenvalue $\mu_k = \frac{k^2 \pi^2}{L^2}. $ And the key insight here is that we can make these as small as we wish simply by taking $L$ arbitrarily large.
On its own, this doesn't answer your question because the interval isn't a compact Riemannian manifold and letting $L$ get large in one dimension corresponds to the volume becoming infinite. However, we can use this idea to construct examples of metrics of fixed volume on other spaces whose $k$-th eigenvalue is very small. To this end, take a manifold of dimension $n \geq 2$ and prescribe a Riemannian metric so that the space is Gromov-Hausdorff close to an interval of length $L$. We can do this by stretching a neighborhood around a point into a long cigar-shaped end.
By taking $L$ large enough but rescaling the rest of the metric, we can fix the volume to be arbitrary. Doing so, we obtain a quasi-isometry $\phi: M \to [0,L]$. If we do this construction in an appropriate way, the eigenfunctions $u_k of $(M,g) are well-approximated by the Neumann eigenfunctions of the interval. In other words, we have that $u_k(x) \approx v_k \circ \phi^{-1}(x)$. To see why this is the case, it's helpful to think about the following picture, which shows how this can be done for a three-holed torus.
We have picked the metric so that the ends of $M$ and all of its handles (i.e., the non-trivial topology) occupt regions where $v_k \circ \phi^{-1}(x)$ is essentially constant. As such, the contribution to the Rayleigh quotient in these regions is essentially negligible. Furthermore, the Neumann conditions $v_k \circ \phi^{-1}(x)$ ensure that we can define $u_k$ smoothly at the tips of the ends. But in the ends (which occupy the bulk of the space), the metric is essentially just $[0,L] \times S^{n-1}$ so the approximation works well.
To convert this into a rigorous proof, you use this construction to build a cigar-shaped end on the manifold and then construct functions which are constant in the original manifold but oscillate through the end. Doing so carefully, you obtain a collection of functions which are mutually orthogonal but whose Rayleigh quotients are arbitrarily small.